Add ability to run broker steps synchronously or asynchronously

[rpappalax]

porting over feature request from loads-broker: loads/loads-broker#102

Currently steps are asynchronous. so, if you want to run them asynchronously, you can cheat the system by adding a staggered interval "run_delay" to each consecutive step. To prevent overlap, I usually also add in an additional 30 second delay. This has the same effect as running each step back-to-back (more or less), but is a bit cumbersome to calculate and re-calculate the run delays when you wish to increment the step run durations.

The other side-effect of this is that the broker assumes you need all the instances for each step up front (which isn't true if you're running the steps back-to-back). For example, if I have 3 steps that require 1, 2 and 3 attack nodes (per step) respectively, the broker sets up 6 nodes up front. This isn't a big deal for small attack swarms, but can get close to AWS quotas for larger swarms if we don't manage this more efficiently.

Example scalability test:

{
  "name": "Shavar Server Testing",
  "plans": [

    {
      "name": "loadtest-shavar",
      "description": "4 boxes",
      "steps": [
        {
          "name": "******************** RUN #01 *************************",
          "instance_count": 6,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        },
        {
          "name": "******************** RUN #02 *************************",
          "run_delay": 330,
          "instance_count": 10,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        },
        {
          "name": "******************** RUN #03 *************************",
          "run_delay": 660,
          "instance_count": 14,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        },
        {
          "name": "******************** RUN #04 *************************",
          "run_delay": 990,
          "instance_count": 18,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        },
        {
          "name": "******************** RUN #05 *************************",
          "run_delay": 1320,
          "instance_count": 22,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        },
        {
          "name": "******************** RUN #06 *************************",
          "run_delay": 1650,
          "instance_count": 24,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        },
        {
          "name": "******************** RUN #07 *************************",
          "run_delay": 1980,
          "instance_count": 28,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        },
        {
          "name": "******************** RUN #08 *************************",
          "run_delay": 2320,
          "instance_count": 32,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        },
        {
          "name": "******************** RUN #09 *************************",
          "run_delay": 2650,
          "instance_count": 36,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        },
        {
          "name": "******************** RUN #10 *************************",
          "run_delay": 2980,
          "instance_count": 40,
          "instance_region": "us-east-1",
          "instance_type": "m3.medium",
          "run_max_time": 300,
          "container_name": "firefoxtesteng/shavar-loadtests:latest",
          "environment_data": [
            "URL_SERVER=https://shavar.stage.mozaws.net",
            "VERBOSE=",
            "TEST_CONNECTIONS=1",
            "TEST_DURATION=300"
          ],
          "volume_mapping": "/var/log:/var/log/$RUN_ID:rw",
          "docker_series": "shavar_tests"
        }
      ]
    }
  ]
}

[pdehaan]

Yes! 💯

This is what we're currently hacking on for Firefox Send and Voice Fill (formerly "Speak To Me"). It is super easy to tweak some process/worker counts and durations without the need to recalculate run delays and start times.

#!/usr/bin/env sh

readonly DURATION=500
readonly PROCESSES=( 12 18 24 30 36 42 )
readonly WORKERS=( 1 )

for p in "${PROCESSES[@]}"
do
  for w in "${WORKERS[@]}"
  do
    echo "----- Now running w/ (-d $DURATION -p $p -w $w) -----"
    molotov -d $DURATION -cx -p $p -w $w loadtest.py
    echo "zzzzz (30s)"
    sleep 30
  done
done