Parallelize creating cluster snapshot

[macsko]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Creating cluster snapshots using SetClusterState can take a long time, especially in large clusters with plenty of pods. Fortunately, we can parallelize the work done per each node. This PR does it by calling workqueue.ParallelizeUntil capped at 16 workers.

I verified the performance and it gives me near 7 times faster execution for 1k nodes, 40k pods scenario, but it depends on the machine's CPU.

In the future, basic store used for scale down can be also changed similarly.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Adding pods when creating a cluster snapshot for scale up is now parallelized, improving performance. Parallelization can be adjusted or disabled by setting the cluster-snapshot-parallelization flag.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: macsko
Once this PR has been reviewed and has the lgtm label, please assign x13n for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• cluster-autoscaler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[macsko]

/cc @yaroslava-serdiuk @aleksandra-malinowska

[yaroslava-serdiuk]

Wow, cool feature!
Let's make in backwards compatible by introducing a flag. We can default it to 16, but in this case let's add release note since we change the behavior. In case this feature will brake someone (seems unlikely, but still we need to consider all scenarious) it should be easy to disable it (set flag to 1).

[yaroslava-serdiuk]

cc: @x13n

[macsko]

Let's make in backwards compatible by introducing a flag.

Added. If the flag is > 1, the snapshot is created using a new way.

[yaroslava-serdiuk]

Overall LGTM
Thanks! Could you also mention how did you measure performance? I suppose it's a CA loop duration, right?

@x13n Could you take a look?

[x13n]

Thanks for the PR! Out of curiosity, did you test it only on the Go benchmarks or in an actual cluster? I wonder how much it improves in practice.

[macsko]

I wonder how much it improves in practice.

Yes, I tested it, but forgot to write. Before the change, in real cluster, for 1k nodes and 40k pods scale, a single SetClusterState took 300-500ms. With this improvement it takes 40-100ms.