Tools / Datadog Interview questions

Real User Monitoring captures actual visitor sessions on your web or mobile app - page loads, clicks, navigation, resource timing, and errors - directly from the browser or device, rather than inferring user experience from backend metrics alone.

A small SDK embedded in the frontend collects these events and sends them to Datadog, where they can be broken down by device, browser, geography, or app version to see who is actually affected by a slow page or a crash.

Because RUM events carry the same session and trace identifiers used elsewhere in Datadog, a slow page load can be linked directly to the backend request and traces that caused it.

Database Monitoring (DBM) is a Datadog product that gives query-level visibility into databases like PostgreSQL, MySQL, and SQL Server - showing which specific queries are slow, their execution plans, and what they're waiting on.

It's collected by a lightweight database integration configured on the Agent, which samples query activity and execution plans directly from the database's own introspection views rather than requiring full, expensive query logging to be turned on.

This goes well beyond a generic CPU or connection-count graph, letting you see exactly which query is responsible for a spike in database load.

Network Performance Monitoring (NPM) shows network traffic between hosts, containers, and services - connections, throughput, retransmits, and latency - tagged the same way as the rest of your Datadog data, so you can see which service-to-service connection is actually degraded.

It's collected using an eBPF-based System Probe bundled with the Agent, which observes TCP connection behavior at the kernel level without requiring any change to application code.

This complements host-level infrastructure metrics by answering a different question: not just 'is this host healthy' but 'is the network path between these two specific services healthy.'

Serverless Monitoring extends Datadog's metrics, traces, and logs to serverless functions - most commonly AWS Lambda, but also Azure Functions and Google Cloud Functions - where there's no host to install a traditional Agent on.

It's typically enabled via a Lambda layer/extension (or the Datadog Forwarder) that runs alongside the function, capturing invocation metrics, cold start duration, and traces with minimal added latency to the function itself.

Because serverless functions are short-lived and billed per invocation, this monitoring specifically surfaces serverless-relevant signals like cold starts, timeouts, and throttles that don't apply the same way to a long-running host.

The Cluster Agent is a specialized Kubernetes-aware component that sits between the Kubernetes API server and the per-node Datadog Agents, acting as a single, centralized point of contact for cluster-level data instead of every node querying the API server directly.

It relays cluster metadata (like pod labels and ownership) down to node Agents so they can enrich locally collected metrics, and it powers cluster-wide features like Cluster Checks, Horizontal Pod Autoscaling on custom metrics, and the Admission Controller.

Running one Cluster Agent (typically as a small Deployment) alongside the per-node Agent DaemonSet reduces load on the Kubernetes API server and centralizes RBAC permissions that would otherwise need to be granted to every node.

CI Visibility gives observability into continuous integration pipelines - build and test durations, failure rates, and flaky tests - across providers like Jenkins, GitLab, CircleCI, and GitHub Actions, in one consistent view.

It ingests pipeline and job-level events plus individual test results, letting teams see whether a slowdown is isolated to one job, one branch, or spread across the whole pipeline.

A commonly cited use case is surfacing flaky tests - tests that pass and fail inconsistently without code changes - which are otherwise hard to spot from CI logs alone.

Error Tracking automatically detects, groups, and deduplicates errors coming from logs, APM traces, and RUM sessions into a single trackable issue, instead of leaving every occurrence as a separate, unrelated log line.

Each issue shows a stack trace, first-seen and last-seen timestamps, occurrence count, and the affected service, so a spike in one underlying bug is immediately visible as one thing getting worse, not a wall of similar-looking log entries.

Issues can be assigned, linked to a monitor, or triaged directly from the Error Tracking view, cutting down the manual correlation work of finding the same error scattered across different log lines.

Continuous Profiler is an always-on, low-overhead code profiler that runs in production, capturing where CPU time, memory allocations, and lock contention are actually being spent inside your application's code - down to the specific function or line.

Unlike a one-off profiling session, it runs continuously with sampling-based collection (rather than tracing every single call), which keeps overhead low enough to leave enabled in production all the time.

Profiles are correlated with APM traces, so a slow endpoint in the trace view can be linked directly to the exact code path consuming the most CPU during that request.

Incident Management is Datadog's built-in tooling for declaring, coordinating, and documenting an incident from detection through resolution, without needing a separate standalone tool.

Declaring an incident creates a structured record with severity, an assigned commander, and a live timeline, and can automatically provision a dedicated Slack channel and notify relevant on-call responders.

As responders investigate, relevant graphs, monitor state, and comments get captured on the timeline, which then feeds directly into a postmortem document once the incident is resolved.

Cloud Cost Management ingests billing data from your cloud providers (like an AWS Cost and Usage Report) and correlates it with the same tags already used across your infrastructure monitoring, so cost can be broken down by team, service, or environment.

It surfaces where spend is concentrated and flags common sources of waste, such as idle or oversized compute resources, right alongside the performance data for those same resources.

Because it reuses existing infrastructure tags rather than requiring a separate cost-tagging exercise, cost visibility stays consistent with whatever tagging scheme is already in place for monitoring.

An API key identifies your Datadog organization and authorizes data to be submitted to it - it's what the Agent and integrations use to send metrics, traces, and logs in.

An application key, by contrast, is tied to an individual user and their permissions, and is used alongside an API key to authenticate programmatic actions taken on behalf of that user, such as creating a dashboard or querying data via the API.

In short: the API key gets data in, while the application key controls what a specific user or script is allowed to do once talking to the API.

The Datadog Terraform provider lets you define dashboards, monitors, synthetic tests, notebooks, and other Datadog resources as version-controlled configuration files, applied through standard Terraform workflows instead of manual UI changes.

This makes Datadog configuration reviewable through normal pull requests, repeatable across multiple environments or accounts, and auditable since changes go through the same change-management process as any other infrastructure.

It's commonly used to keep dev, staging, and production monitoring consistent, and to prevent configuration drift that happens when changes are made ad hoc through the UI by different people over time.

An outlier monitor flags a single host, container, or other entity that's behaving statistically different from its peer group - for example, one host in a fleet of fifty with unusually high latency while the rest look normal.

Rather than requiring a fixed, manually chosen threshold, it compares each group member against the others using a statistical algorithm, which is useful when 'normal' varies across a fleet and there's no single absolute number that fits every host.

This makes it well suited for catching the 'one bad instance in a large, otherwise-uniform group' pattern that a plain threshold monitor would either miss or constantly false-alarm on.

A forecast monitor projects a metric's future values based on its historical pattern - using linear or seasonal forecasting algorithms - and alerts when the predicted value is likely to breach a configured threshold, not just when the current value already has.

This is commonly used for capacity planning, such as predicting that disk usage will hit 100% in roughly ten days at the current growth rate, giving teams lead time to act before an actual outage.

Because it relies on historical trend, it needs enough consistent history to forecast reliably, and works best on metrics with a fairly steady or seasonal growth pattern rather than highly erratic ones.

The Service Catalog is a centralized registry of every service running across your organization, showing ownership, on-call contacts, dependencies, documentation links, and a health or quality score for each one.

It's populated either from YAML definition files checked into a service's repository or automatically detected from existing APM and infrastructure tags, so it doesn't necessarily require a separate manual registration step.

It's especially valuable at scale: when an incident hits an unfamiliar service, the catalog answers 'who owns this and how do I reach them' in seconds instead of a scramble through chat history or wikis.

Datadog can ingest telemetry (traces, metrics, and logs) produced using the vendor-neutral OpenTelemetry standard, either by sending data directly to the Agent's built-in OTLP receiver or by routing it through an OpenTelemetry Collector configured with the Datadog exporter.

This means teams already instrumented with OpenTelemetry SDKs don't have to rip out and replace that instrumentation with Datadog-specific tracing libraries to get the same Datadog UI experience.

Once ingested, OpenTelemetry data is mapped into Datadog's internal model, so it shows up in the same Trace Explorer, service pages, and dashboards as data from Datadog's own tracing libraries.

An Agent flare is a diagnostic bundle - generated with a single command - that packages the Agent's logs, configuration (with secrets automatically redacted), and status information into a single archive.

It's the standard way to hand troubleshooting context to Datadog support: instead of manually copy-pasting logs and configs, a flare captures a consistent, complete snapshot in one step and can be uploaded directly or attached to a support ticket.

Because secrets like API keys are scrubbed automatically before packaging, it's generally safe to generate and share without manually editing the output first.

Sensitive Data Scanner inspects telemetry - most commonly logs - for patterns matching sensitive data, such as credit card numbers, social security numbers, or custom-defined patterns, using either Datadog's built-in pattern library or custom regular expressions.

When a match is found, the configured rule can redact the value, hash it, or simply flag/tag it, and this scanning happens as part of the processing pipeline before or during indexing, not as an afterthought once data is already searchable.

It's commonly used to help meet compliance requirements by ensuring sensitive values never end up stored in plain text in Datadog, even if an upstream application accidentally logs them.

Workflow Automation is a low-code tool for building automated sequences of actions - chaining API calls, notifications, and approvals - typically triggered by a monitor firing or an incident being declared.

A workflow might, for example, automatically restart a service, post an update to an incident's Slack channel, and open a ticket, all from one alert, without an engineer manually performing each step.

Because workflows are built visually from reusable building blocks rather than custom scripts, they lower the barrier for teams to automate repetitive operational toil without maintaining a separate scripting codebase.

Application Security Management (ASM) detects and can block application-layer attacks - like SQL injection or cross-site scripting - by observing requests at runtime using the same tracing instrumentation already deployed for APM.

Because it reuses existing APM instrumentation rather than requiring a separate agent, security findings appear correlated with the same traces, services, and infrastructure context engineers already use for performance debugging.

It also covers vulnerability detection in application dependencies (open-source libraries), surfacing known CVEs affecting the exact services running them, tied to that service's real traffic and exposure.

An API key authorizes data to flow into a Datadog organization - it's org-scoped, used by the Agent and integrations, and by itself doesn't grant any particular user's permissions for reading or managing resources.

An application key is scoped to an individual user and inherits that user's role and permissions; it's required alongside an API key for most read/write API calls, such as programmatically creating a monitor or querying a dashboard's data.

Practically, this means revoking a departing employee's application key cuts off their personal API access without disrupting the Agent's org-wide API key still used to submit data, while rotating the API key affects every integration relying on it at once.

The node-level Agent runs as a DaemonSet, one pod per node, collecting host, pod, and container metrics, logs, and traces local to that specific node - it's the workhorse doing per-node data collection.

The Cluster Agent runs as a small, typically single or low-replica Deployment and centralizes cluster-wide concerns: it's the sole component querying the Kubernetes API server for cluster metadata, dispatches Cluster Checks (checks that should run once cluster-wide, not once per node), and powers Horizontal Pod Autoscaling on custom/external metrics.

This split means node Agents only need RBAC permissions scoped to their own node's kubelet, not the whole API server, which is what actually reduces both API server load and the security surface of the per-node Agent fleet.

Downtime silences a monitor's notifications for a defined time window - useful during planned maintenance - while leaving the monitor itself fully active: it keeps evaluating, keeps recording history, and simply suppresses the alert from actually notifying anyone during that window.

Deleting a monitor removes it entirely, along with its evaluation history, which means you lose the ability to review what happened during the muted period afterward, and you'd have to recreate the exact same configuration from scratch to monitor that condition again.

Downtime is also easily scheduled and automatically expires, whereas a deleted monitor requires someone to remember to manually recreate it once maintenance ends - a common source of monitoring gaps if forgotten.

Standard log-based error monitoring treats every error occurrence as its own separate log line - to understand whether an error is new, worsening, or recurring, someone has to manually search and eyeball the pattern across many individual entries.

Error Tracking automatically groups and deduplicates those occurrences into a single issue based on stack trace and error signature, tracking occurrence count, first-seen/last-seen timestamps, and affected service as one continuously updating record instead of scattered lines.

This turns 'search logs for this error message' into 'look at this issue's trend,' which is materially faster during triage, especially for errors with slightly varying messages (like different IDs embedded in the text) that a plain log search might otherwise treat as unrelated.

Rather than instrumenting every function call (which would add meaningful latency to every request), the profiler uses statistical sampling - periodically capturing a snapshot of what the application's threads are doing, at a configurable interval - and reconstructs a representative picture of where time is spent from those samples over time.

Collected samples are batched and sent asynchronously in the background, off the request-processing path, so profiling doesn't add latency to the actual requests being served.

This keeps typical overhead in the low single-digit percent range, which is what makes it practical to leave running continuously in production rather than only during dedicated, short-lived profiling sessions.

Use a threshold monitor when you know the exact acceptable value in advance and it's the same for every entity being monitored - like 'any host with disk usage over 90% is a problem,' regardless of how its peers are doing.

Use an outlier monitor when 'normal' genuinely differs across a group and what matters is relative deviation from peers, not an absolute number - like catching one host in a fleet of similar hosts whose latency has drifted noticeably higher than the rest, even if that latency alone wouldn't cross any fixed threshold.

A practical tell: if you'd have to keep manually tuning a threshold per-host to avoid false alarms because 'normal' varies so much between them, that's usually a sign an outlier monitor is the better fit for that specific metric.

A process monitor tracks whether a specific process (matched by name or command-line pattern) is running on a host, and can alert on it stopping unexpectedly or on that process's own resource usage - it's fundamentally about a single process's presence and behavior on one host.

A network monitor, built on Network Performance Monitoring data, alerts on connection-level signals between hosts or services - retransmits, connection failures, or elevated latency on a specific network path - it's about the health of communication between entities, not any single process in isolation.

In practice they answer different failure questions: a process monitor answers 'is my application still running,' while a network monitor answers 'can my services actually reach each other reliably.'

There are two common ingestion paths: sending OTLP (OpenTelemetry Protocol) data directly to the Datadog Agent's built-in OTLP receiver, or routing it through a self-managed OpenTelemetry Collector configured with the Datadog exporter, which then forwards it to Datadog's intake.

Either way, Datadog's backend translates the incoming OpenTelemetry spans, metrics, and logs into its own internal representation, mapping OpenTelemetry semantic conventions (like service.name) onto Datadog's tagging model so the data appears in the same Trace Explorer and service pages as natively instrumented data.

Choosing between the two paths is mostly about where you want the translation and any additional processing to happen: going straight to the Agent's OTLP receiver is simpler for a single-Agent setup, while a Collector-based pipeline gives more flexibility to add processors (like batching or filtering) before data ever reaches Datadog.

By default, every tag combination submitted on a custom metric becomes a separate, billed timeseries - so a high-cardinality tag can inflate cost dramatically even if nobody ever actually queries by that tag.

Metrics without Limits decouples ingestion from indexing: all the data you send continues to be ingested and available in aggregate, but you choose which specific tags stay indexed (queryable), and billing for that metric is based on the indexed volume, not the full ingested volume.

This lets teams keep submitting rich, high-cardinality data from their application code without changing it, while controlling cost purely through tag configuration on the Datadog side - useful when the application team and the person optimizing cost aren't the same people, and re-instrumenting code isn't practical.

The scanner applies whichever action the matching rule is configured with: redact (replace the sensitive value with a placeholder), hash (replace it with a non-reversible hash, preserving the ability to spot repeated occurrences without exposing the value), or simply flag/tag the event without altering its content.

This happens as part of processing, before the log is fully indexed, so a redaction action means the sensitive value is prevented from ever being stored in plain text in the searchable index - not cleaned up after the fact.

Because rules can be scoped to specific log sources or namespaces, teams can apply stricter handling (like automatic redaction) to logs known to be higher risk while leaving lower-risk sources untouched or only flagged for awareness.

It ingests detailed billing export data from cloud providers, such as an AWS Cost and Usage Report, which breaks charges down to the individual resource level rather than a single lump sum per account.

That resource-level billing data is then joined against the tags already applied to those same resources - team, service, environment - so cost can be sliced along the same dimensions engineers already use for performance monitoring, without a separate cost-tagging project.

Resources that are untagged, or inconsistently tagged, typically fall into an 'unallocated' or 'untagged' bucket in the resulting cost breakdown, which is itself a useful signal pointing at where tagging hygiene needs improvement before attribution can be fully accurate.

Sampling rules operate earlier, at the client/Agent level, determining what percentage of traces are even generated and sent to Datadog in the first place - they control ingestion volume and cost, and also affect the trace-based metrics computed from what's sampled.

Retention filters operate later, after ingestion, deciding which already-ingested traces get indexed (kept searchable) for the full 15-day retention window - Datadog's built-in Intelligent Retention Filter already keeps a representative, error/latency-biased sample by default, and custom retention filters let you guarantee specific business-critical traces stay searchable regardless of that default.

In practice, tune sampling rules when you need to control the raw volume and cost of what's captured at all, and add custom retention filters when you need to guarantee that specific, already-ingested traces - for a critical endpoint, say - remain searchable for the full window rather than relying only on the default intelligent selection.

A Service Level Indicator (SLI) is the actual measured signal - for example, the ratio of successful requests to total requests over some period - it's a number reflecting what's really happening.

A Service Level Objective (SLO) is the target applied to that indicator - for example, '99.9% of requests should succeed over a rolling 30 days' - it's the goal, not the measurement itself.

In Datadog specifically, you configure an SLO by picking (or building) an underlying SLI - either a metric-based ratio query or a monitor's uptime - and layering a target percentage and time window on top of it; the SLI is the raw ingredient, the SLO is the recipe applied to it.

Fleet Automation gives a centralized view of every Agent's version, configuration, and health across an entire fleet, so operators don't have to manually check each host individually to know what's out of date.

It leverages Datadog's remote configuration capability to push upgrades and configuration changes to Agents directly, without requiring an operator to SSH into every host or re-run a configuration management tool across the fleet for routine changes.

Because rollouts can typically be staged or scoped rather than applied everywhere simultaneously, teams can validate an upgrade on a smaller subset of hosts before pushing it fleet-wide, reducing the blast radius if something goes wrong.

As the number of services grows into the hundreds, no single engineer can reliably remember who owns what, which services depend on which, or who to page - information that used to live in someone's head or an out-of-date wiki now needs a system of record.

The Service Catalog centralizes that information (ownership, on-call, dependencies, documentation) and keeps it closer to accurate by sourcing it from the same infrastructure and APM data already flowing in, rather than a separate document nobody remembers to update.

During an incident specifically, this materially reduces time-to-mitigation: instead of a Slack thread asking 'does anyone know who owns this service,' a responder can look it up directly and page the right team immediately.

Browser RUM, via a JavaScript SDK, tracks page views, resource load timing, and browser-specific quality signals like Core Web Vitals - metrics that are meaningful in the context of a web page's lifecycle.

Mobile RUM, via native SDKs for iOS, Android, React Native, or Flutter, tracks screen views, app launch/startup time, and importantly, native app crashes - a concept that doesn't map cleanly onto a browser tab, since a crashed browser tab behaves very differently from a crashed native app.

Both feed into the same RUM product and can correlate with backend APM traces the same way, but the specific vocabulary of events and quality metrics reflects the very different runtime environments of a web page versus an installed app.

Logs sent to a cloud storage archive (rather than fully indexed) aren't searchable in the Log Explorer by default - rehydration is the process of pulling a specific subset back out of that archive, based on a time range and optional filter, into a temporary, searchable index.

This lets teams keep the bulk of their log volume in cheap, unsearched archival storage most of the time, and only pay the cost of making a slice of it searchable on the rare occasions - an investigation, an audit request - when it's actually needed.

Because rehydration re-indexes real data on demand, it takes some time to complete and incurs its own indexing cost for that rehydrated slice, so it's a deliberate, scoped action rather than something done casually for every archived log.

Use dashboards-as-code (via the Terraform provider or the dashboards API with exported JSON) when a dashboard needs to be reviewed through a pull request, replicated identically across multiple environments or accounts, or kept in sync with an infrastructure-as-code workflow that already governs everything else.

The UI editor remains the faster path for exploratory, one-off dashboards during an active investigation, where the overhead of a commit-review-apply cycle would slow down getting an answer, and where the dashboard may not need to persist or be replicated anywhere.

A common pattern is prototyping quickly in the UI, then exporting the finished layout into version control once it's proven useful enough to standardize on and share across teams or environments.

When a page loads (or a mobile app launches), the RUM SDK initializes and starts a new session, generating a unique session ID that will tag every event produced during that session's lifetime.

As the user interacts with the app - navigating pages, clicking, waiting on network requests - the SDK captures each of these as structured RUM events (view, action, resource, error) locally, buffering them rather than sending each one individually as it happens.

If the backend is also instrumented with Datadog APM (or OpenTelemetry ingested by Datadog), the SDK injects trace context into outgoing network requests, allowing the resulting backend trace to be linked back to the specific RUM session and view that triggered it.

The SDK periodically flushes its buffered events to Datadog's RUM intake over HTTPS, batching them to reduce request overhead rather than making a network call per individual event; if Session Replay is enabled, DOM snapshots and mutations are captured and sent through a related but separate replay pipeline.

Once ingested, RUM events are subject to Datadog's own retention/sampling rules for RUM (which can differ from APM's), and a default cross-product retention filter can additionally guarantee that traces for a small percentage of sessions remain indexed on the APM side, so a given session's user-facing view and its backend trace can both actually be inspected together.

sequenceDiagram
  participant SDK as RUM SDK (browser/mobile)
  participant App as Backend service (APM)
  participant DD as Datadog RUM intake
  SDK->>SDK: Init session, generate session ID
  SDK->>SDK: Capture view/action/resource/error events
  SDK->>App: Network request (trace context injected)
  App-->>SDK: Response
  SDK->>DD: Periodic batched flush of buffered events
  DD->>DD: Apply RUM sampling/retention, cross-product APM link

Start from the fact that ingestion and indexing are billed differently: sampling controls how much is generated/ingested at all, while retention filters control what stays searchable for the 15-day window - the default Intelligent Retention Filter already keeps errors and a diverse latency sample without configuration, and its spans don't count against indexed-span billing.

Layer deliberate custom retention filters on top only for traces you specifically need guaranteed, complete visibility into beyond what intelligent sampling would naturally catch - for example, 100% retention on a small set of business-critical endpoints (checkout, payment) rather than broadly indexing everything.

Avoid the common mistake of creating many overlapping custom filters that all capture largely the same traffic; each additional filter adds to indexed volume independently, so redundant filters compound cost without adding proportional value - periodically reviewing filter overlap against actual query patterns keeps this in check.

Combine this with tuned ingestion sampling for very high-volume, low-value traffic (like health checks) so the base ingested volume itself stays lean, rather than relying on retention filters alone to control cost after the fact - the two levers work best used together, not as substitutes for each other.

First confirm the database-level prerequisite: DBM relies on database introspection views and settings (for example, ensuring query text capture is enabled, like track_activity_query_size on PostgreSQL) - if these aren't enabled at the database level, the Agent has nothing to read regardless of its own configuration.

Check that the monitoring user configured for the integration actually has the required permissions - most engines need a role with monitoring-specific privileges (such as PostgreSQL's pg_monitor) rather than full administrative access; an under-privileged user will connect successfully for basic checks but silently fail to pull query-level data.

Verify the Agent's integration configuration explicitly has DBM collection enabled (it's typically a distinct flag from the base database check, not automatically on just because the base integration works) and that the config version matches what the installed database engine actually supports.

Confirm basic connectivity and authentication are actually fine by checking the base (non-DBM) database check status first - if the standard health/connection check is failing too, DBM's absence is a symptom of that broader connectivity problem, not a DBM-specific issue.

Finally, check the Agent's own logs and status output for the specific database check, since DBM-related collection errors (permission denied on a specific view, version incompatibility) are usually logged explicitly there rather than failing completely silently.

CWS uses eBPF probes attached to relevant kernel syscalls - process execution, file access, network activity - to observe what's actually happening on a host or inside a container at runtime, with low overhead since the observation happens in-kernel rather than through expensive userspace interception of every call.

Observed events are streamed to a CWS module within the local Agent, which enriches each event with process ancestry (the chain of parent processes that led to it) and container/orchestration context (which pod, which image), since raw syscall data alone rarely tells you enough about intent or origin.

Enriched events are evaluated against detection rules - Datadog ships default, curated rulesets (for common attack patterns like privilege escalation or suspicious file access) and also supports custom rules written for specific environments and threat models.

When a rule matches, CWS generates a signal that's sent to Datadog, carrying the full enriched context (process tree, container, host) needed to investigate the event without having to separately reconstruct that context after the fact from raw logs.

Because detection happens locally at the kernel level in near-real time rather than through periodic scanning, CWS can catch and surface active runtime threats as they happen, complementing posture management tools that instead look for static misconfigurations before anything has actually run.

flowchart TD
  A[eBPF probes on syscalls] --> B[Local CWS module: enrich with process ancestry + container context]
  B --> C{Match detection rule?}
  C -- Yes --> D[Generate signal with full context]
  C -- No --> A
  D --> E[Send to Datadog for investigation]

Identify checks that are redundant when run per-node - typically checks against an external endpoint, a load-balanced Kubernetes Service, or a managed external database - since running the same check independently on every node Agent wastes resources checking the exact same target repeatedly with no added value.

Configure those specific checks as Cluster Checks so the Cluster Agent dispatches each one to run exactly once across the cluster, rather than once per node, which both reduces redundant load on the checked target and lowers overall Agent resource consumption cluster-wide.

For clusters where check volume itself is high enough to strain the node Agents dispatched to run them, deploy dedicated Cluster Check Runners - a small, separate pool of Agents whose only job is running cluster checks - so that regular per-node endpoint checks aren't competing for the same Agent resources as cluster-wide checks.

Keep genuinely node-local checks (like a container runtime check, or a check on a service that's only meaningfully checked from the node it lives on) as regular per-node Autodiscovery checks rather than Cluster Checks - converting truly node-specific checks to cluster-wide ones would actually lose the node-specific context that made them useful in the first place.

An incident is declared - either manually by a responder or automatically triggered from a monitor - creating a structured record with an initial severity, a designated commander, and an auto-provisioned communication channel (commonly a dedicated Slack channel) to centralize discussion.

As the incident progresses, the timeline auto-populates with relevant context: monitor state changes, graphs pulled in by responders, and manually added comments, building a chronological record of what was observed and done without requiring anyone to separately write it all down after the fact.

The incident moves through defined states - typically something like Active, then Stable once impact is contained but not fully resolved, then Resolved once the underlying issue is actually fixed - and severity can be adjusted as understanding of impact evolves during the response.

Throughout, responders can attach related resources - the triggering monitor, relevant dashboards, or a notebook - directly to the incident record, so the full investigative context stays linked rather than scattered across separate tools.

Once resolved, the accumulated timeline feeds into a postmortem template, pre-populated with the incident's actual chronology, which teams then annotate with root cause analysis and follow-up action items - turning documentation that would otherwise be reconstructed from memory into something largely already captured in real time.

flowchart LR
  A[Incident declared: manual or monitor-triggered] --> B[Commander assigned, channel provisioned]
  B --> C[Timeline auto-populates: monitors, graphs, comments]
  C --> D{State}
  D -- Active --> C
  D -- Stable --> C
  D -- Resolved --> E[Postmortem generated from timeline]

This isn't really an either-or choice in practice - they operate at different speeds and with different strengths, and the strongest approach uses Watchdog RCA to shortcut the slowest part of manual investigation rather than replacing manual judgment entirely.

Watchdog RCA automatically correlates anomalies across related services and infrastructure using AI, surfacing a likely root cause hypothesis within seconds of an incident starting - work that would otherwise require a human to manually cross-reference dashboards, logs, and recent deploys across potentially many services, which is exactly the slow, high-cognitive-load part of an incident that most delays MTTR.

Manual analysis remains essential for validating that hypothesis against business context Watchdog doesn't have - whether a correlated signal is actually causal versus coincidental, or how a novel failure mode not resembling anything in its training data should be interpreted - AI correlation is a strong first pass, not a substitute for domain judgment on unfamiliar failures.

In practice, the fastest MTTR path in complex, multi-service incidents is to use Watchdog RCA's hypothesis as an immediate starting point to focus the investigation, then have a human responder confirm or redirect from there, rather than starting entirely from scratch or blindly trusting an automated hypothesis without validation.

First confirm the receiving side is actually configured to accept OTLP: check that the Agent's OTLP receiver (or the Collector's Datadog exporter) is enabled on the expected port and protocol (gRPC vs HTTP), since a protocol or port mismatch between the SDK's exporter config and the receiver is one of the most common silent failure points.

Check the OpenTelemetry SDK's own export configuration for batching and shutdown behavior - if the application exits (or is killed) before the SDK's batch span processor flushes its buffer, in-flight spans can be lost with no error surfaced anywhere, which looks identical to spans never having been created at all.

Verify network connectivity along the actual path from the application to wherever it's exporting - directly to the Agent, or to an intermediate Collector - since spans failing to export due to a network or firewall issue typically fail quietly from the application's perspective unless export errors are explicitly logged.

Check for missing or malformed resource attributes, especially service.name - Datadog's mapping from OpenTelemetry's semantic conventions to its own service/tagging model depends on these being present and well-formed, and spans with missing critical attributes can be dropped or mis-tagged during translation rather than simply appearing under an unexpected service name.

Finally, rule out sampling: confirm what sampling configuration is active in the OpenTelemetry SDK itself (a head-based sampler could be dropping spans well before they ever reach the exporter), since that's a separate layer from anything configured on the Datadog side.

For logs, scanning happens as a stage within the log processing pipeline: after earlier processors (like the Grok Parser) have extracted structured fields, configured scanning rules are evaluated against the log's content and matching attributes, and any configured action - redact, hash, or flag - is applied before the log proceeds to indexing.

Because this happens before indexing, a redaction decision is final in the sense that the original sensitive value never becomes part of the searchable, stored record - there's no separate 'clean up afterward' step needed, which is what makes it suitable for compliance-sensitive data.

For APM, a comparable scanning capability can be applied to span tags and resource data, evaluated as spans are processed, so sensitive values that might otherwise leak into trace metadata (like a request parameter accidentally tagged onto a span) get the same treatment before that span data is stored.

Rule scope matters operationally: rules can be applied broadly across an entire org or narrowed to specific log sources, services, or namespaces, letting teams apply strict, low-false-negative rules to known higher-risk sources (like payment-related logs) while avoiding unnecessary processing overhead or false positives on unrelated telemetry.

flowchart LR
  A[Log arrives at pipeline] --> B[Grok Parser extracts fields]
  B --> C[Sensitive Data Scanner evaluates rules]
  C --> D{Match?}
  D -- Yes --> E[Apply action: redact/hash/flag]
  D -- No --> F[Pass through unchanged]
  E --> G[Proceed to indexing]
  F --> G

Prefer the Lambda extension/layer approach for instrumentation over a heavier wrapper that fully re-initializes tracing logic on every cold start, since the extension runs as a lightweight, separate process alongside the function and adds materially less to init duration than a fully synchronous wrapping approach.

Use the built-in cold-start metric from the Lambda integration specifically (rather than inferring it from duration alone) to track cold-start frequency and duration as its own signal, since blending cold and warm invocation durations together in one metric can hide a cold-start problem inside an otherwise-fine average.

Where cold start latency is genuinely business-critical (a user-facing, latency-sensitive endpoint), recognize that Datadog's monitoring can measure and alert on the problem but can't architecturally fix it - the actual mitigation, such as AWS Provisioned Concurrency, has to be configured at the AWS level; Datadog's role here is giving you the visibility to know it's needed and to verify it's working afterward.

Finally, watch for instrumentation itself adding to cold start time as libraries and dependencies grow - periodically reviewing the deployed package size and the specific libraries pulled in by tracing instrumentation is worth doing, since bloated deployment packages are a common, easy-to-overlook contributor to slow cold starts that has nothing to do with the monitoring vendor at all.

Each Agent maintains a persistent, authenticated connection (or regular polling cadence) to Datadog's remote configuration backend, checking for configuration updates that apply to it based on its own tags and current configuration state.

When an operator pushes a change - enabling a new product, updating APM sampling rules, or rolling out a security ruleset - through the UI, API, or a tool like Fleet Automation, that change is published as a signed configuration update targeted at the relevant scope of Agents, rather than requiring a new deployment artifact.

Agents receiving an applicable update validate its signature before applying it, which protects the update mechanism itself from being used to push unauthorized or tampered configuration to a fleet of Agents with wide infrastructure access.

Once validated, the Agent applies the change locally - often without needing a full restart for many configuration types - which is what allows near-real-time, staged rollouts across potentially thousands of hosts without coordinating a traditional deployment pipeline for what is, functionally, just a configuration change.

This same mechanism underlies capabilities like Fleet Automation's staged Agent upgrades and dynamically updated APM sampling or security rules, since all of them are, at the transport level, the same signed-update-plus-local-validation pattern applied to different kinds of configuration payloads.

Start by checking the completeness of the underlying billing export itself - confirm the cloud provider's cost and usage report (or equivalent) is generating complete files without gaps, since a partial or delayed export will show as a genuine drop in attributed cost that has nothing to do with Datadog's tagging logic at all.

Check for a timing mismatch: cloud providers typically finalize detailed billing data with some lag, so cost figures for the most recent day or two may look artificially low or incomplete simply because the underlying data hasn't fully landed yet, not because attribution logic is actually wrong.

Audit tagging consistency on the resources themselves - inconsistent or missing tags (a resource tagged Team in one place and team elsewhere, or simply untagged) will cause that resource's cost to fall into an unallocated bucket rather than the team it should logically belong to, and this is the single most common root cause of attribution looking 'wrong.'

For shared or pooled resources (like a shared Kubernetes cluster running workloads for multiple teams), check how the allocation rule for that shared cost is actually configured - an unexpected default allocation method (evenly split versus usage-proportional, for instance) can look like a data error when it's actually a configuration choice about how shared cost should be divided.

Finally, cross-check a specific discrepancy against the raw cloud provider billing console directly for the same resource and time window, to isolate whether the mismatch originates in the cloud provider's own billing data or in Datadog's processing/attribution of it - that distinction determines whether the fix belongs in cloud tagging hygiene or in the Cloud Cost Management configuration itself.