Idea: Throw error/warning message about bad config in Graphite querier pod logs

[Cimbel]

What is the problem you are trying to solve?

When I deployed Mimir using mimir-distributed helm-chart in Kubernetes, I found that there was an error in Graphite querirer config(in default values graphite.querier.storageSchemas.schemasConfiguration ) which caused production metrics that are older than 8 days disapear. There weren't any logs of Graphite querier pod that would tell that issue occured in config.

Which solution do you envision (roughly)?

I want Graphite querier pod to throw at least warning message about bad config in logs. Because there was a typo and we spent 2 month root casue of the issue. It ended up asking Grafana Labs for the help. So, I don't want this to happen again neither to me nor to other people that using mimir-distributed helm chart to deploy Mimir on Kubernetes platform with Graphite enabled.

Have you considered any alternatives?

No response

Any additional context to share?

Platform: Kubernetes
Service: Mimir (Graphite enabled)

How long do you think this would take to be developed?

Not sure

What are the documentation dependencies?

No response

Proposer?

No response

[ywwg]

Upon examination, mimir can read files with this syntax of duration units and therefore will not throw an error when it encounters them. However, carbon-relay-ng, the component that sends data to Grafana, is more strict and does not support the longer names for units.

[ywwg]

Making mimir more strict could have the unintended consequence of breaking many configuration files that use the more verbose units and are handled fine. The solution is probably best applied in crng. I'll leave this bug open for now in case we come up with a different approach.