Align code with error message and raise timeout after 30 min

[albertvillanova]

Align code with error message and raise timeout after 30 min.

Note current error message says:

lighteval/src/lighteval/models/endpoints/endpoint_model.py

Line 258 in 2ef9740

"Endpoint did not start within 30 minutes, there was a timeout. Please inspect the logs."

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[clefourrier]

Hmm, just re-read the code, and the maximum amount of time is managed by MAX_TIME_FOR_SPINUP.
At the place you pointed, we wait for 10 min for an endpoint to load, and we either

• try to scale it up if it failed
• retry it as such if it's still pending (it's in the big for loop).

This allows to wait in total for 30 min

[albertvillanova]

I have re-read the code and I think there is an error in its logic:

• The first time we enter in the while loop, the endpoint is launched and waited with a 10 min timeout
• If the endpoint is not created within 10 min, an InferenceEndpointTimeoutError is raised
• This error is caught in the except clause, which logs an error message and re-raises
• There is no other outer except clause, and therefore we exit the while loop by raising the error

[clefourrier]

You're right, the raise e should be removed so that it tries to spin up the endpoint with the same params for an extra 10 min

[clefourrier]

                except InferenceEndpointTimeoutError as e:
                    logger.error(
                        "Endpoint did not start within 30 minutes, there was a timeout. Please inspect the logs."
                    )
                    raise e

should be

                except InferenceEndpointTimeoutError as e:
                    logger.warning(
                        "Endpoint did not start within 10 minutes, there was a timeout. Trying for 10 extra minutes, or stopping if we passed 30 min."
                    )

[albertvillanova]

@clefourrier with the suggestion above, no error will be raised even after 30 min timeout.

[albertvillanova]

Let me try to simplify the logic as much as possible so that we are sure it works as expected.

[albertvillanova]

While refactoring, I also got confused about the must_scaleup_endpoint variable: if I understand correctly, whenever an InferenceEndpointError is raised, must_scaleup_endpoint is set to True, even if the inference point already existed and was set to be reused.
Is this expected?