Running the mnist example with SMAC tuner results in an error #3788

AkshayRoyal · 2020-10-08T15:23:45Z

AkshayRoyal
Oct 8, 2020

Environment:

NNI version: 1.8
NNI mode (local|remote|pai):local
Client OS:Fedora32
Server OS (for remote mode only):
Python version: 3.8.5
PyTorch/TensorFlow version: TensorFlow2.3.0
Is conda/virtualenv/venv used?: conda
Is running in Docker?:No

Log message:

nnimanager.log:
[10/8/2020, 8:33:21 PM] INFO [ 'Datastore initialization done' ] [10/8/2020, 8:33:21 PM] INFO [ 'RestServer start' ] [10/8/2020, 8:33:21 PM] INFO [ 'Construct local machine training service.' ] [10/8/2020, 8:33:21 PM] INFO [ 'RestServer base port is 8080' ] [10/8/2020, 8:33:21 PM] INFO [ 'Rest server listening on: http://0.0.0.0:8080' ] [10/8/2020, 8:33:22 PM] INFO [ 'NNIManager setClusterMetadata, key: local_config, value: {"useActiveGpu":true}' ] [10/8/2020, 8:33:22 PM] INFO [ 'Specified GPU indices: undefined' ] [10/8/2020, 8:33:22 PM] INFO [ 'NNIManager setClusterMetadata, key: trial_config, value: {"command":"python3 mnist.py","codeDir":"/home/fu-workstation/Downloads/nni/examples/trials/mnist-tfv2/.","gpuNum":1}' ] [10/8/2020, 8:33:22 PM] INFO [ 'required GPU number is 1' ] [10/8/2020, 8:33:22 PM] INFO [ 'Starting experiment: rnlQhc8m' ] [10/8/2020, 8:33:22 PM] INFO [ 'Change NNIManager status from: INITIALIZED to: RUNNING' ] [10/8/2020, 8:33:22 PM] INFO [ 'Add event listeners' ] [10/8/2020, 8:33:22 PM] INFO [ 'Run local machine training service.' ] [10/8/2020, 8:33:22 PM] WARNING [ 'gpu_metrics file does not exist!' ] [10/8/2020, 8:33:27 PM] ERROR [ 'Dispatcher error: This socket has been ended by the other party' ] [10/8/2020, 8:33:27 PM] ERROR [ 'Error: Dispatcher stream error, tuner may have crashed.\n at EventEmitter.dispatcher.onError (/home/fu-workstation/anaconda3/envs/tf23/nni/core/nnimanager.js:509:32)\n at EventEmitter.emit (events.js:182:13)\n at Socket.IpcInterface.outgoingStream.on (/home/fu-workstation/anaconda3/envs/tf23/nni/core/ipcInterface.js:41:72)\n at Socket.emit (events.js:182:13)\n at Socket.writeAfterFIN [as write] (net.js:405:8)\n at IpcInterface.sendCommand (/home/fu-workstation/anaconda3/envs/tf23/nni/core/ipcInterface.js:48:38)\n at NNIManager.pingDispatcher (/home/fu-workstation/anaconda3/envs/tf23/nni/core/nnimanager.js:358:29)' ] [10/8/2020, 8:33:27 PM] INFO [ 'Change NNIManager status from: RUNNING to: ERROR' ] [10/8/2020, 8:33:27 PM] WARNING [ 'Commands jammed in buffer!' ]
dispatcher.log:

[10/08/2020, 08:33:23 PM] INFO (smac_AutoML/Thread-1) update search space in SMAC.
[10/08/2020, 08:33:23 PM] INFO (smac_AutoML/Thread-1) SMAC call: /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/__main__.py --exp_params eyJhdXRob3JOYW1lIjoiZGVmYXVsdCIsImV4cGVyaW1lbnROYW1lIjoiZXhhbXBsZV9tbmlzdF9ib2hiIiwidHJpYWxDb25jdXJyZW5jeSI6MSwibWF4RXhlY0R1cmF0aW9uIjozNjAwMCwibWF4VHJpYWxOdW0iOjEwMDAsInRyYWluaW5nU2VydmljZVBsYXRmb3JtIjoibG9jYWwiLCJ0dW5lciI6eyJidWlsdGluVHVuZXJOYW1lIjoiU01BQyIsImNsYXNzQXJncyI6eyJvcHRpbWl6ZV9tb2RlIjoibWF4aW1pemUiLCJjb25maWdfZGVkdXAiOnRydWV9fSwidmVyc2lvbkNoZWNrIjp0cnVlLCJjbHVzdGVyTWV0YURhdGEiOlt7ImtleSI6ImNvZGVEaXIiLCJ2YWx1ZSI6Ii9ob21lL2Z1LXdvcmtzdGF0aW9uL0Rvd25sb2Fkcy9ubmkvZXhhbXBsZXMvdHJpYWxzL21uaXN0LXRmdjIvLiJ9LHsia2V5IjoiY29tbWFuZCIsInZhbHVlIjoicHl0aG9uMyBtbmlzdC5weSJ9XX0=
[10/08/2020, 08:33:23 PM] PRINT INFO:	Reading scenario file: scenario.txt

[10/08/2020, 08:33:23 PM] PRINT INFO:	Output to smac3-output_2020-10-08_20:33:23_055107

What issue meet, what's expected?:
When I try to run SMAC tuner by using the following config.yml file(check in the reproduce error section). The status of the nni dashboard changes to ERROR

How to reproduce it?:

In the nni examples run mnist-tfv2 with the below config.yml file

experimentName: example_mnist_bohb
trialConcurrency: 1
maxExecDuration: 10h
maxTrialNum: 1000
#choice: local, remote, pai
trainingServicePlatform: local
searchSpacePath: search_space.json
#choice: true, false
useAnnotation: false
tuner:
  builtinTunerName: SMAC
  classArgs:
    optimize_mode: maximize
    config_dedup: True
trial:
  command: python3 mnist.py
  codeDir: .
  gpuNum: 1
localConfig:
  useActiveGpu: True

Additional information:

Answered by J-shang

Nov 30, 2020

@AkshayRoyal @inspurwusixuan We found that this is due to the use of swig 4.0 in these Linux release versions, a temporary solution is to downgrade the swig version and reinstall pyrfr.

The following issue automl/SMAC3#545 from the original SMAC3 repo may help you.

View full answer

QuanluZhang · 2020-10-09T03:05:22Z

QuanluZhang
Oct 9, 2020

@AkshayRoyal , I have tried your configuration and failed to reproduce your problem. It is likely that SMAC has not been successfully installed in your system.

First, please check the output of this command nnictl package install --name=SMAC for installing SMAC. Is there error message there?

Second, it is strange that tuner is crashed, but there is no error message in dispatcher.log. Could you run nnictl log stderr <exp-id> to check whether there is error message there?

0 replies

AkshayRoyal · 2020-10-09T03:29:40Z

AkshayRoyal
Oct 9, 2020
Author

@QuanluZhang . This is the output for nnictl package install --name=SMAC

  Cloning https://github.com/QuanluZhang/ConfigSpace.git to /tmp/pip-req-build-dyp8rglt
Requirement already satisfied (use --upgrade to upgrade): ConfigSpaceNNI==0.4.7 from git+https://github.com/QuanluZhang/ConfigSpace.git in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from -r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1))
Collecting git+https://github.com/QuanluZhang/SMAC3.git (from -r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2))
  Cloning https://github.com/QuanluZhang/SMAC3.git to /tmp/pip-req-build-2beoi7to
Requirement already satisfied (use --upgrade to upgrade): smac==0.8.0 from git+https://github.com/QuanluZhang/SMAC3.git in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from -r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2))
Requirement already satisfied: numpy in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from ConfigSpaceNNI==0.4.7->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1)) (1.19.1)
Requirement already satisfied: pyparsing in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from ConfigSpaceNNI==0.4.7->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: typing in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from ConfigSpaceNNI==0.4.7->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1)) (3.7.4.3)
Requirement already satisfied: Cython in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from ConfigSpaceNNI==0.4.7->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1)) (0.29.21)
Requirement already satisfied: setuptools in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (49.6.0.post20200814)
Requirement already satisfied: scipy>=0.18.1 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.5.2)
Requirement already satisfied: six in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.15.0)
Requirement already satisfied: psutil in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (5.7.2)
Requirement already satisfied: pynisher>=0.4.1 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.6.1)
Requirement already satisfied: scikit-learn>=0.18.0 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.23.2)
Requirement already satisfied: pyrfr>=0.5.0 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.8.0)
Requirement already satisfied: sphinx in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (3.2.1)
Requirement already satisfied: sphinx_rtd_theme in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.5.0)
Requirement already satisfied: joblib in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.16.0)
Requirement already satisfied: docutils>=0.3 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from pynisher>=0.4.1->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.16)
Requirement already satisfied: threadpoolctl>=2.0.0 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from scikit-learn>=0.18.0->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.1.0)
Requirement already satisfied: sphinxcontrib-qthelp in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.3)
Requirement already satisfied: alabaster<0.8,>=0.7 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.7.12)
Requirement already satisfied: sphinxcontrib-serializinghtml in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.1.4)
Requirement already satisfied: requests>=2.5.0 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.24.0)
Requirement already satisfied: sphinxcontrib-devhelp in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.2)
Requirement already satisfied: Pygments>=2.0 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.7.1)
Requirement already satisfied: packaging in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (20.4)
Requirement already satisfied: Jinja2>=2.3 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.11.2)
Requirement already satisfied: snowballstemmer>=1.1 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.0.0)
Requirement already satisfied: sphinxcontrib-htmlhelp in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.3)
Requirement already satisfied: babel>=1.3 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.8.0)
Requirement already satisfied: sphinxcontrib-applehelp in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.2)
Requirement already satisfied: imagesize in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.2.0)
Requirement already satisfied: sphinxcontrib-jsmath in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.1)
Requirement already satisfied: idna<3,>=2.5 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from requests>=2.5.0->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from requests>=2.5.0->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.25.10)
Requirement already satisfied: chardet<4,>=3.0.2 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from requests>=2.5.0->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from requests>=2.5.0->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2020.6.20)
Requirement already satisfied: MarkupSafe>=0.23 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from Jinja2>=2.3->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.1.1)
Requirement already satisfied: pytz>=2015.7 in ./anaconda3/envs/tf23/lib/python3.8/site-packages (from babel>=1.3->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2020.1)
Building wheels for collected packages: ConfigSpaceNNI, smac
  Building wheel for ConfigSpaceNNI (setup.py) ... done
  Created wheel for ConfigSpaceNNI: filename=ConfigSpaceNNI-0.4.7-cp38-cp38-linux_x86_64.whl size=4300769 sha256=7f0d8a83f9d80dbfeabdcefc3232c0f18e9c54b5de47472a17a20283bf325774
  Stored in directory: /tmp/pip-ephem-wheel-cache-wqgaf4zc/wheels/ee/77/ac/d55b9b65449fb09ca265beddcb8f33c10bb557e964b6b05cea
  Building wheel for smac (setup.py) ... done
  Created wheel for smac: filename=smac-0.8.0-py3-none-any.whl size=144817 sha256=d93b5acf9a3ae2e06f4e9f085ffe6b7fb9ebc39f4936d491feedb1c0bea8b796
  Stored in directory: /tmp/pip-ephem-wheel-cache-wqgaf4zc/wheels/36/1e/b2/791232ed991a9fe2d1932108a8fd0a3fde5d6d0a23d01d8536
Successfully built ConfigSpaceNNI smac
ERROR: name SMAC already installed
ERROR: installation failed!

And this is the output for nnictl log stderr

  self.re = re.compile(self.reString)
-----------------------------------------------------------------------
                Experiment start time 2020-10-09 08:55:36
-----------------------------------------------------------------------
Error: Dispatcher stream error, tuner may have crashed.
    at EventEmitter.dispatcher.onError (/home/fu-workstation/anaconda3/envs/tf23/nni/core/nnimanager.js:509:32)
    at EventEmitter.emit (events.js:182:13)
    at Socket.IpcInterface.outgoingStream.on (/home/fu-workstation/anaconda3/envs/tf23/nni/core/ipcInterface.js:41:72)
    at Socket.emit (events.js:182:13)
    at Socket.writeAfterFIN [as write] (net.js:405:8)
    at IpcInterface.sendCommand (/home/fu-workstation/anaconda3/envs/tf23/nni/core/ipcInterface.js:48:38)
    at NNIManager.pingDispatcher (/home/fu-workstation/anaconda3/envs/tf23/nni/core/nnimanager.js:358:29)```

0 replies

AkshayRoyal · 2020-10-09T03:43:15Z

AkshayRoyal
Oct 9, 2020
Author

I also tried removing the SMAC package using nnictl package uninstall SMAC which successfully removed SMAC package.
Then I tried installing SMAC using nnictl package install --name SMAC which gave this output

(tf23) [fu-workstation@localhost mnist-tfv2]$ nnictl package install --name SMAC
Collecting git+https://github.com/QuanluZhang/ConfigSpace.git (from -r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1))
  Cloning https://github.com/QuanluZhang/ConfigSpace.git to /tmp/pip-req-build-09fx0qnu
Requirement already satisfied (use --upgrade to upgrade): ConfigSpaceNNI==0.4.7 from git+https://github.com/QuanluZhang/ConfigSpace.git in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from -r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1))
Collecting git+https://github.com/QuanluZhang/SMAC3.git (from -r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2))
  Cloning https://github.com/QuanluZhang/SMAC3.git to /tmp/pip-req-build-jem4936a
Requirement already satisfied (use --upgrade to upgrade): smac==0.8.0 from git+https://github.com/QuanluZhang/SMAC3.git in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from -r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2))
Requirement already satisfied: numpy in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from ConfigSpaceNNI==0.4.7->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1)) (1.19.1)
Requirement already satisfied: pyparsing in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from ConfigSpaceNNI==0.4.7->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: typing in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from ConfigSpaceNNI==0.4.7->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1)) (3.7.4.3)
Requirement already satisfied: Cython in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from ConfigSpaceNNI==0.4.7->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 1)) (0.29.21)
Requirement already satisfied: setuptools in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (49.6.0.post20200814)
Requirement already satisfied: scipy>=0.18.1 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.5.2)
Requirement already satisfied: six in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.15.0)
Requirement already satisfied: psutil in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (5.7.2)
Requirement already satisfied: pynisher>=0.4.1 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.6.1)
Requirement already satisfied: scikit-learn>=0.18.0 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.23.2)
Requirement already satisfied: pyrfr>=0.5.0 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.8.0)
Requirement already satisfied: sphinx in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (3.2.1)
Requirement already satisfied: sphinx_rtd_theme in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.5.0)
Requirement already satisfied: joblib in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.16.0)
Requirement already satisfied: docutils>=0.3 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from pynisher>=0.4.1->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.16)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from scikit-learn>=0.18.0->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.1.0)
Requirement already satisfied: sphinxcontrib-devhelp in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.2)
Requirement already satisfied: sphinxcontrib-applehelp in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.2)
Requirement already satisfied: sphinxcontrib-htmlhelp in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.3)
Requirement already satisfied: requests>=2.5.0 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.24.0)
Requirement already satisfied: snowballstemmer>=1.1 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.0.0)
Requirement already satisfied: alabaster<0.8,>=0.7 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (0.7.12)
Requirement already satisfied: Jinja2>=2.3 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.11.2)
Requirement already satisfied: sphinxcontrib-jsmath in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.1)
Requirement already satisfied: Pygments>=2.0 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.7.1)
Requirement already satisfied: packaging in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (20.4)
Requirement already satisfied: sphinxcontrib-qthelp in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.0.3)
Requirement already satisfied: sphinxcontrib-serializinghtml in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.1.4)
Requirement already satisfied: babel>=1.3 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.8.0)
Requirement already satisfied: imagesize in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.2.0)
Requirement already satisfied: idna<3,>=2.5 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from requests>=2.5.0->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from requests>=2.5.0->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2020.6.20)
Requirement already satisfied: chardet<4,>=3.0.2 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from requests>=2.5.0->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from requests>=2.5.0->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.25.10)
Requirement already satisfied: MarkupSafe>=0.23 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from Jinja2>=2.3->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (1.1.1)
Requirement already satisfied: pytz>=2015.7 in /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages (from babel>=1.3->sphinx->smac==0.8.0->-r /home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/nni/smac_tuner/requirements.txt (line 2)) (2020.1)
Building wheels for collected packages: ConfigSpaceNNI, smac
  Building wheel for ConfigSpaceNNI (setup.py) ... done
  Created wheel for ConfigSpaceNNI: filename=ConfigSpaceNNI-0.4.7-cp38-cp38-linux_x86_64.whl size=4300755 sha256=fef0b516cab0879af3928b067dc7ae88d276119c78cb4235085e26b166446ce5
  Stored in directory: /tmp/pip-ephem-wheel-cache-ud_01zjy/wheels/ee/77/ac/d55b9b65449fb09ca265beddcb8f33c10bb557e964b6b05cea
  Building wheel for smac (setup.py) ... done
  Created wheel for smac: filename=smac-0.8.0-py3-none-any.whl size=144817 sha256=0a353f248e5fce170dd51cd5a2612936df48b41bdca668c90e3acbacf5c9bd7d
  Stored in directory: /tmp/pip-ephem-wheel-cache-ud_01zjy/wheels/36/1e/b2/791232ed991a9fe2d1932108a8fd0a3fde5d6d0a23d01d8536
Successfully built ConfigSpaceNNI smac
SMAC installed!

Also, I checked with nnictl package list and the output was

nnictl package list
+-----------------+------------+-----------+--------=-------------+------------------------------------------+
|      Name       |    Type    | Installed |      Class Name      |               Module Name                |
+-----------------+------------+-----------+----------------------+------------------------------------------+
| SMAC            | tuners     | Yes       | SMACTuner            | nni.smac_tuner.smac_tuner                |
| PPOTuner        | tuners     | No        | PPOTuner             | nni.ppo_tuner.ppo_tuner                  |
| BOHB            | advisors   | Yes       | BOHB                 | nni.bohb_advisor.bohb_advisor            |
+-----------------+------------+-----------+----------------------+------------------------------------------+

After that I reran nnictl create --config config.yml which resulted in

nnictl create --config config.yml 
INFO:  expand searchSpacePath: search_space.json to /home/fu-workstation/Downloads/nni/examples/trials/mnist-tfv2/search_space.json 
INFO:  expand codeDir: . to /home/fu-workstation/Downloads/nni/examples/trials/mnist-tfv2/. 
/home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/pyparsing.py:3190: FutureWarning: Possible set intersection at position 3
  self.re = re.compile(self.reString)
INFO:  Starting restful server...
INFO:  Successfully started Restful server!
INFO:  Setting local config...
INFO:  Successfully set local config!
INFO:  Starting experiment...
INFO:  Successfully started experiment!
------------------------------------------------------------------------------------
The experiment id is IsZUHonS
The Web UI urls are: http://127.0.0.1:8080   http://192.168.1.3:8080   http://192.168.122.1:8080
------------------------------------------------------------------------------------

You can use these commands to get more information about the experiment
------------------------------------------------------------------------------------
         commands                       description
1. nnictl experiment show        show the information of experiments
2. nnictl trial ls               list all of trial jobs
3. nnictl top                    monitor the status of running experiments
4. nnictl log stderr             show stderr log content
5. nnictl log stdout             show stdout log content
6. nnictl stop                   stop an experiment
7. nnictl trial kill             kill a trial job by id
8. nnictl --help                 get help information about nnictl
------------------------------------------------------------------------------------
Command reference document https://nni.readthedocs.io/en/latest/Tutorial/Nnictl.html
------------------------------------------------------------------------------------

where the config.yml file contained this

authorName: NNI Example
experimentName: MNIST TF v2.x
trialConcurrency: 1
maxExecDuration: 1h
maxTrialNum: 10
trainingServicePlatform: local  # choices: local, remote, pai
searchSpacePath: search_space.json
useAnnotation: false
tuner:
    builtinTunerName: SMAC   # choices: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner,
                            #          GPTuner, SMAC (SMAC should be installed through nnictl)
    classArgs:
        optimize_mode: maximize  # choices: maximize, minimize
trial:
  command: python3 mnist.py
  codeDir: .
  gpuNum: 1
localConfig:
  useActiveGpu: True

This is how I was again able to reproduce the issue.
Also, I forgot to mention that I was able to run other tuners and advisors too but SMAC didn't work.

0 replies

QuanluZhang · 2020-10-09T05:54:57Z

QuanluZhang
Oct 9, 2020

@AkshayRoyal , thanks for the detailed test and description. The failure might be induced by fedora system. Could you run the command python3 -m unittest test_builtin_tuners.BuiltinTunersTestCase.test_smac under the directory src/sdk/pynni/tests to check the output info of the test?

0 replies

AkshayRoyal · 2020-10-09T06:39:30Z

AkshayRoyal
Oct 9, 2020
Author

Hi @QuanluZhang. Thanks for your prompt response.
The output for the command you mentioned is

(tf23) [fu-workstation@localhost tests]$ python3 -m unittest test_builtin_tuners.BuiltinTunersTestCase.test_smac
[10/09/2020, 12:06:10 PM] WARNING (nni.protocol) IPC pipeline not exists, maybe you are importing tuner/assessor from trial code?
/home/fu-workstation/anaconda3/envs/tf23/lib/python3.8/site-packages/pyparsing.py:3190: FutureWarning: Possible set intersection at position 3
 self.re = re.compile(self.reString)
INFO:smac_AutoML:update search space in SMAC.
INFO:smac_AutoML:SMAC call: python3 -m unittest test_builtin_tuners.BuiltinTunersTestCase.test_smac
Segmentation fault (core dumped)

0 replies

QuanluZhang · 2020-10-10T01:34:21Z

QuanluZhang
Oct 10, 2020

@AkshayRoyal , we reproduced your problem in a fedora docker container. We are checking whether we can make it work.

0 replies

AkshayRoyal · 2020-10-25T09:55:39Z

AkshayRoyal
Oct 25, 2020
Author

Hi @QuanluZhang, Is there any update regarding the issue?

0 replies

inspurwusixuan · 2020-10-27T23:51:09Z

inspurwusixuan
Oct 27, 2020

I got the same error when I was using SMAC tuner. Checked that SMAC was successfully installed and was using Ubuntu 20.04.

0 replies

J-shang · 2020-11-30T05:01:55Z

J-shang
Nov 30, 2020
Maintainer

@AkshayRoyal @inspurwusixuan We found that this is due to the use of swig 4.0 in these Linux release versions, a temporary solution is to downgrade the swig version and reinstall pyrfr.

The following issue automl/SMAC3#545 from the original SMAC3 repo may help you.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running the mnist example with SMAC tuner results in an error #3788

{{title}}

Replies: 9 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Running the mnist example with SMAC tuner results in an error #3788

AkshayRoyal Oct 8, 2020

Replies: 9 comments

QuanluZhang Oct 9, 2020

AkshayRoyal Oct 9, 2020 Author

AkshayRoyal Oct 9, 2020 Author

QuanluZhang Oct 9, 2020

AkshayRoyal Oct 9, 2020 Author

QuanluZhang Oct 10, 2020

AkshayRoyal Oct 25, 2020 Author

inspurwusixuan Oct 27, 2020

J-shang Nov 30, 2020 Maintainer

AkshayRoyal
Oct 8, 2020

QuanluZhang
Oct 9, 2020

AkshayRoyal
Oct 9, 2020
Author

AkshayRoyal
Oct 9, 2020
Author

QuanluZhang
Oct 9, 2020

AkshayRoyal
Oct 9, 2020
Author

QuanluZhang
Oct 10, 2020

AkshayRoyal
Oct 25, 2020
Author

inspurwusixuan
Oct 27, 2020

J-shang
Nov 30, 2020
Maintainer