enable_virtual_node_label config not work

[chenlujjj]

Describe the bug

I set enable_virtual_node_label: true in the service graph config, but the metrics still don't have virtual_node label

To Reproduce

1. Setup a distributed tempo
2. Enable service graph metrics generator and remote write metrics to TSDB
3. Set enable_virtual_node_label: true in the service graph config
4. Query metrics via traces_service_graph_request_total{connection_type="virtual_node"}

Expected behavior

The metrics should have virtual_node label as the doc (https://grafana.com/docs/tempo/latest/metrics-generator/service_graphs/#activate-enable_virtual_node_label) said

Environment:

• Infrastructure: [Kubernetes]
• Deployment tool: [helm]

Additional Context

[joe-elliott]

For the virtual node label to be populated a few other things have to be true. You can see the code here that is checking additional details on the spans:

tempo/modules/generator/processor/servicegraphs/servicegraphs.go

Lines 397 to 420 in f00ed6a

	if len(e.ClientService) == 0 {
	// If the client service is not set, it means that the span could have been initiated by an external system,
	// like a frontend application or an engineer via `curl`.
	// We check if the span we have is the root span, and if so, we set the client service to "user".
	if _, parentSpan := parseKey(e.Key()); len(parentSpan) == 0 {
	e.ClientService = "user"
	if p.Cfg.EnableVirtualNodeLabel {
	e.Dimensions[virtualNodeLabel] = "client"
	}
	p.onComplete(e)
	}
	} else if len(e.ServerService) == 0 && len(e.PeerNode) > 0 {
	// If client span does not have its matching server span, but has a peer attribute present,
	// we make the assumption that a call was made to an external service, for which Tempo won't receive spans.
	e.ServerService = e.PeerNode
	if p.Cfg.EnableVirtualNodeLabel {
	e.Dimensions[virtualNodeLabel] = "server"
	}
	p.onComplete(e)
	}

And we have some docs that describe these details as well:

https://grafana.com/docs/tempo/latest/metrics-generator/service_graphs/#virtual-nodes