Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI execution cannot load dependencies, GPU is not used #1259

Open
HG4554 opened this issue Aug 18, 2022 · 1 comment
Open

CLI execution cannot load dependencies, GPU is not used #1259

HG4554 opened this issue Aug 18, 2022 · 1 comment

Comments

@HG4554
Copy link

HG4554 commented Aug 18, 2022

Describe the bug
FaceSwap GUI generates CLI commands that cannot utilize the GPU due to Tensorflow dependency errors. The Conda env has to be "activated" for all dependencies to load correctly, it is no longer sufficient to directly invoke Python by it's path in the Faceswap env. This did work in the past.

I believe this is the underlying cause of other issues like #1244, hat tip to Replican for posting the solution in the forum.

To Reproduce
Steps to reproduce the behavior:

  1. Configure the Extract tool and run from the Faceswap GUI, it will run normally and utilize the GPU.
  2. Click Generate and run the CLI equivalent in a terminal. It will print dependency errors and run on CPU only.

Expected behavior
If GPU works in GUI, it should work in CLI.

Desktop (please complete the following information):

  • OS: AlmaLinux 9 (RHEL 9)
  • Python 3.9.12
  • Conda Version 4.12.0
  • Commit ID ee25a31 (current master)

Additional context

This is the current invocation that fails to use GPU:

ec2-user@ip-172-30-1-104 /m/data > /mnt/data/homedir/faceswap8/miniconda3/envs/faceswap/bin/python /mnt/data/homedir/faceswap8/faceswap/faceswap.py e
xtract -i /mnt/data/sourcedata/test/test.mp4 -o /mnt/data/sourcedata/test/output -al /mnt/data/sourcedata/test/test.fsa -D s3fd -A fan -nm none -rf 0 -min 0 -
l 0.4 -sz 512 -een 1 -si 0 -ssf -L VERBOSE     
...
2022-08-18 17:32:51.796104: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.
so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /mnt/data/homedir/faceswap8/miniconda3/envs/faceswap/lib/python3.9/site-pack
ages/cv2/../../lib64:
2022-08-18 17:32:51.796132: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries 
mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup 
the required libraries for your platform.
Skipping registering GPU devices...

This invocation activates the Conda env first, then runs the same CLI command which does utilize the GPU:

ec2-user@ip-172-30-1-104:/mnt/data/homedir/faceswap8$ source miniconda3/etc/profile.d/conda.sh activate
ec2-user@ip-172-30-1-104:/mnt/data/homedir/faceswap8$ conda activate faceswap
(faceswap) ec2-user@ip-172-30-1-104:/mnt/data/homedir/faceswap8$ which python
/mnt/data/homedir/faceswap8/miniconda3/envs/faceswap/bin/python                
(faceswap) ec2-user@ip-172-30-1-104:/mnt/data/homedir/faceswap8$ python /mnt/data/homedir/faceswap8/faceswap/faceswap.py extract -i /mnt/data/sourcedat
a/test/test.mp4 -o /mnt/data/sourcedata/test/output -al /mnt/data/sourcedata/test/test.fsa -D s3fd -A fan -nm none -rf 0 -min 0 -l 0.4 -sz 512 -een 1 -si 0 -s
sf -L VERBOSE                          
...
2022-08-18 17:51:57.393937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13650 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5
@HG4554 HG4554 changed the title CLI generation invokes Conda wrong, GPU is not used CLI execution cannot load dependencies, GPU is not used Aug 18, 2022
@torzdf
Copy link
Collaborator

torzdf commented Aug 18, 2022

I'm trying to think of the use-case for this... it may be that the documentation needs to be updated. I haven't tested the above, but then I have never attempted to run faceswap from outside of the Conda environment.

It probably helps that I wrote the setup script, so know that it adds LD_LIBRARY_PATH to the activate script:

def _set_env_vars(self) -> None:

Ultimately, I'm surprised that just executing the python binary worked in the past, as the Conda Env holds more than just a python virtual environment (required Cuda/cuDNN binaries and the like).

Generating the cli arguments from the GUI is just a convenience function. Adapting it to test for Conda environments and the like is most likely a bit out of scope.

I'm not sure if it's the same as the Docker issue (I do not use Docker), as theoretically Docker has it's own globally installed Cuda/cuDNN which should be being utilized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants