The OpenTelemetry collector failed to export data because the ingester became unavailable.

[edgarkz]

Describe the bug
Hello,
Tempo ingester shutdown is causing trace spans to be dropped due to failed otel collector export operations.

2024-12-25T03:58:10.168Z	error	exporterhelper/queue_sender.go:97	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unknown desc = Ingester is shutting down", "dropped_items": 1483}
go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
	go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:97
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
	go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:57
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1

To Reproduce
Steps to reproduce the behavior:

1. Start Tempo (SHA or version)
2. Perform Operations (Read/Write/Others)

Expected behavior
Running in distributed mode with multiple ingesters and distributors should not cause upstream errors of ingesters not avail.

Environment:

• Infrastructure: eks k8s 1.29
• Deployment tool: helm distributed 2.6
• tempo components are running on spot instances

Additional Context
Otel collector version - aws-otel-collector:v0.38.2