Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a metric of AppWrappers counts per state #675

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add a metric of AppWrappers counts per state #675

wants to merge 4 commits into from

Conversation

ronensc
Copy link

@ronensc ronensc commented Oct 31, 2023 •
edited
Loading

Issue link

Closes #674

What changes have been made

This PR adds a custom metric to expose how many AppWrappers are per state.
It is built on top of #651. To review only the delta, skip the first 3 commits in this PR.

Verification steps

Same as mentioned in #651, I checked that the metrics are available in the codeflare-operator operator by running it:

# in codeflare-operator root
go mod edit -replace github.com/project-codeflare/multi-cluster-app-dispatcher=../multi-cluster-app-dispatcher
make build
NAMESPACE=default go run ./main.go -kubeconfig <path-to-kubeconfig>

and in a new shell:

curl localhost:8080/metrics | grep appwrappers_count


# HELP mcad_appwrappers_count AppWrappers count per state
# TYPE mcad_appwrappers_count gauge
mcad_appwrappers_count{state="Completed"} 0
mcad_appwrappers_count{state="Deleted"} 0
mcad_appwrappers_count{state="Failed"} 0
mcad_appwrappers_count{state="Pending"} 0
mcad_appwrappers_count{state="Running"} 1
mcad_appwrappers_count{state="RunningHoldCompletion"} 0

To see the counts change, I created a sample AppWrapper:

# in MCAD root
kubectl apply -f doc/usage/aw-01.yaml

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • Testing is not required for this change

@ronensc
Add initial metrics
8f96ada 
Add dependency: sigs.k8s.io/controller-runtime v0.14.6

go get sigs.k8s.io/[email protected]
go mod tidy
@ronensc
Update metrics once a minute
7713655
@ronensc
Fix typo
2a4a551
@ronensc
Add queue metrics
0a5954a
@openshift-ci
Copy link

openshift-ci bot commented Oct 31, 2023

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign asm582 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ronensc
ronensc commented Oct 31, 2023
View reviewed changes
pkg/controller/queuejob/metrics.go
awList, err := controller.appWrapperLister.List(labels.Everything())

if err != nil {
klog.Errorf("[updateQueue] Unable to obtain the list of AppWrappers: %+v", err)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case of error, should we update the counters to 0 or leave them as is?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should leave them as is and retry to update with new value.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This update is done periodically. So instead of retrying, we can simply wait for the next update on the next tick.
I dug deeper into the implementation of controller.appWrapperLister.List(labels.Everything()), and it seems that since labels.Everything().Empty() returns true, there is no way that controller.appWrapperLister.List() will return an error. So the discussion seems to be purely theoretical.

@ronensc ronensc mentioned this pull request Nov 1, 2023
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mamy-CS mamy-CS mamy-CS left review comments

@anishasthana anishasthana Awaiting requested review from anishasthana

@metalcycling metalcycling Awaiting requested review from metalcycling

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Expose a metric of AppWrappers counts per state
2 participants
@ronensc @mamy-CS