Fail to schedule on available workers (typically occurs when a bad job is submitted.)

[mimran-stripe]

We're seeing some behavior where when a bad job is submitted (e.g. if it is crashing due to to a NullPointerException) other jobs are not getting adequately scheduled. Our list of possible workers renders ~0 -- Our "DEFAULT_CLUSTER" returns the following response:

{                                                                                                                                                                          
  "numRegisteredTaskExecutors": 58,                                                                                                                                        
  "numAvailableTaskExecutors": 0,                                                                                                                                          
  "numOccupiedTaskExecutors": 28,                                                                                                                                          
  "numAssignedTaskExecutors": 0,                                                                                                                                           
  "numDisabledTaskExecutors": 30                                                                                                                                           
}           

but we certainly have less than 28 instances attempting to spin up.

We usually see this behavior after we've determined one job is crashlooping (crashes, and Mantis attempts to reschedule).

We've been able to resolve this in the past by manually restarting instances of mantis-server-agent. Do you have any suggestions on how to avoid this failure?

[Andyz26]

@mimran-stripe how does your agent behave when a job is unassigned/crashed? Two things worth looking at here:

• is the jobActor on the control plane submitting new workers to replace the crashed one?
• were crashed workers/agents recovered and re-registered with the control plane as available?