Prev Next

Tools / Datadog Interview questions

How do you troubleshoot missing spans from an OpenTelemetry-instrumented service?

First confirm the receiving side is actually configured to accept OTLP: check that the Agent's OTLP receiver (or the Collector's Datadog exporter) is enabled on the expected port and protocol (gRPC vs HTTP), since a protocol or port mismatch between the SDK's exporter config and the receiver is one of the most common silent failure points.

Check the OpenTelemetry SDK's own export configuration for batching and shutdown behavior - if the application exits (or is killed) before the SDK's batch span processor flushes its buffer, in-flight spans can be lost with no error surfaced anywhere, which looks identical to spans never having been created at all.

Verify network connectivity along the actual path from the application to wherever it's exporting - directly to the Agent, or to an intermediate Collector - since spans failing to export due to a network or firewall issue typically fail quietly from the application's perspective unless export errors are explicitly logged.

Check for missing or malformed resource attributes, especially service.name - Datadog's mapping from OpenTelemetry's semantic conventions to its own service/tagging model depends on these being present and well-formed, and spans with missing critical attributes can be dropped or mis-tagged during translation rather than simply appearing under an unexpected service name.

Finally, rule out sampling: confirm what sampling configuration is active in the OpenTelemetry SDK itself (a head-based sampler could be dropping spans well before they ever reach the exporter), since that's a separate layer from anything configured on the Datadog side.

A common silent failure point is a mismatch in:
Spans can be lost with no visible error when:

Invest now in Acorns!!! 🚀 Join Acorns and get your $5 bonus!
Acorns Logo

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Earn passively and while sleeping

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Robinhood Logo

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.


Webull Logo

Webull! Receive free stock by signing up using the link: Webull signup.

More Related questions...

What is Real User Monitoring (RUM) in Datadog? What is Datadog Database Monitoring? What is Network Performance Monitoring in Datadog? What is Datadog Serverless Monitoring? Describe the Datadog Cluster Agent? What is Datadog CI Visibility? What is Datadog Error Tracking? What is Continuous Profiler in Datadog? Describe Datadog Incident Management? What is Datadog Cloud Cost Management? What are API keys and application keys in Datadog? What is the Datadog Terraform provider used for? What is an outlier monitor in Datadog? What is a forecast monitor in Datadog? What is the Datadog Service Catalog? Define OpenTelemetry support in Datadog? What is an Agent flare in Datadog? What is Sensitive Data Scanner in Datadog? Describe Datadog Workflow Automation? What is Application Security Management in Datadog? What is the difference between API keys and application keys? How does the Cluster Agent differ from the node-level Datadog Agent? Why do we use monitor mute/downtime instead of deleting a monitor? What is the difference between Error Tracking and standard log-based error monitoring? How does Datadog's Continuous Profiler collect data without high overhead? When should you use an outlier monitor versus a threshold monitor? What is the difference between a process monitor and a network monitor in Datadog? How does Datadog ingest OpenTelemetry data? Why is Metrics without Limits useful for cost control? What happens when Sensitive Data Scanner detects a match? How does Datadog's Cloud Cost Management attribute spend? When should you use APM trace retention filters versus sampling rules? What is the difference between Service Level Indicators and Service Level Objectives? How does Fleet Automation manage Agent upgrades across a fleet? Why is the Service Catalog important for large engineering organizations? What is the difference between mobile RUM and browser RUM? How does log rehydration work from Datadog archives? When should you use dashboards-as-code instead of the UI editor? Explain the execution flow of a RUM session being recorded and ingested? How can you optimize APM costs using retention filters? How do you troubleshoot a Database Monitoring integration reporting no query metrics? Explain the internal working of Cloud Workload Security (CWS)? How can you optimize Kubernetes monitoring using the Cluster Agent's Cluster Checks? Explain the lifecycle of an incident in Datadog Incident Management? Which is better for reducing MTTR: Watchdog RCA or manual root cause analysis, and why? How do you troubleshoot missing spans from an OpenTelemetry-instrumented service? Explain the execution flow of Sensitive Data Scanner across logs and APM? How can you optimize serverless monitoring for Lambda cold starts? Explain the internal working of Datadog's remote configuration feature? How do you troubleshoot inconsistent cost attribution in Cloud Cost Management?
Show more question and Answers...

Golang

Comments & Discussions