ci: e2e tests unmatched events does not work as expected

[kkourt]

In some recent CI failures, it seemed that even though only one event was unmatched, the unmatched JSON output included all events. The only way to diagnose this was to add -test.v to the e2e test args. For reference, I'm attaching the logs afer enabling -test.v and artifacts from the same run.

In the logs, you can see that only one event was missed:

2024-10-31T08:59:08.8668768Z time="2024-10-31T08:59:08Z" level=info msg="UnorderedEventChecker: checking pending 1/1: ProcessExecChecker/otel-demo-opensearch: Process check failed: ProcessChecker: Pod check failed: PodChecker: PodLabels[app.kubernetes.io/component] (app.kubernetes.io/component=loadgenerator) check failed: 'loadgenerator' does not match full 'otel-demo-opensearch'"
2024-10-31T08:59:08.8672504Z time="2024-10-31T08:59:08Z" level=info msg="PROCESS_EXEC:3584 => no match: UnorderedEventChecker: all 1 check(s) failed, continuing"
2024-10-31T08:59:09.0824913Z I1031 08:59:09.081023   13892 rpcchecker.go:249] event checker labelsEventChecker timed out after 10m0s

But the in the unmatched json event, all events are unmatched:

jq -r '.[] | to_entries[] | .value.checkerName' < labelsEventChecker.eventchecker.unmatched.json
otel-demo-grafana
otel-demo-jaeger
otel-demo-otelcol
otel-demo-prometheus
otel-demo-accountingservice
otel-demo-adservice
otel-demo-cartservice
otel-demo-checkoutservice
otel-demo-currencyservice
otel-demo-emailservice
otel-demo-flagd
otel-demo-frauddetectionservice
otel-demo-frontend
otel-demo-frontendproxy
otel-demo-kafka
otel-demo-loadgenerator
otel-demo-paymentservice
otel-demo-productcatalogservice
otel-demo-quoteservice
otel-demo-recommendationservice
otel-demo-redis
otel-demo-shippingservice
otel-demo-opensearch

labelsEventChecker.eventchecker.events.json
labelsEventChecker.eventchecker.log
labelsEventChecker.eventchecker.unmatched.json
4_ubuntu-22.04-arm64 _ labels.txt