raining process fails due to Out of Memory (OOM) error

[linhcentrio]

Issue Title: Training process fails due to Out of Memory (OOM) error

Description

I encountered an Out of Memory (OOM) issue while running a training script on my AlmaLinux 9 server. The training process terminates abruptly, and the kernel logs show that the process was killed by the OOM killer. Despite configuring ZRAM and having sufficient swap space, the issue persists intermittently.

Environment

• Operating System: AlmaLinux 9
• Python Version: (e.g., Python 3.9.2)
• Training Framework: (e.g., TensorFlow/PyTorch)
• GPU: NVIDIA (Driver and CUDA versions match the framework requirements)
• RAM: 31 GiB
• Swap Space:
  • ZRAM: 8 GiB
  • Swapfile: 114 GiB (unused during OOM events)
• Script Command:

  python run.py train -A /home/deepfake/workspace/faceA/... \
  -B /home/deepfake/workspace/faceB/... \
  -m /home/deepfake/workspace/models/... \
  -t phaze-a -b 16 -i 1000000 -D default -r -s 360 -I 10000 \
  -x /home/deepfake/workspace/faceA/... \
  -y /home/deepfake/workspace/faceB/... \
  -z /home/deepfake/workspace/timelapse/... -M -L INFO

Observed Behavior

• The script starts training and runs for a while before abruptly terminating.
• Kernel logs indicate:

  Out of memory: Killed process <PID> (python) total-vm:<...> anon-rss:<...>
  

• Memory monitoring shows the process consuming nearly all available RAM (~31 GiB) before being killed, while swap remains largely unused.

Expected Behavior

The training process should utilize swap space or adjust memory allocations to avoid being killed by the OOM killer.

Troubleshooting Steps Taken

1. Optimized ZRAM configuration:
   • Increased ZRAM size to 8 GiB and adjusted swap priority.
   • Verified ZRAM is active and functioning as swap:

     NAME       TYPE      SIZE USED PRIO
     /dev/zram0 partition   8G   0B    5
     

2. Reduced batch size: Lowered the -b parameter from 16 to 8.
3. Adjusted system memory settings:
   • Configured /proc/sys/vm parameters:

     vm.overcommit_memory = 2
     vm.overcommit_ratio = 50
     vm.swappiness = 60
     

4. Monitored memory usage: Observed high memory consumption during training without significant swap utilization.

Logs

Kernel log snippet:

Out of memory: Killed process <PID> (python) total-vm:69079928kB, anon-rss:31427696kB, file-rss:65664kB, shmem-rss:40512kB, UID:0 pgtables:73224kB oom_score_adj:0

Steps to Reproduce

1. Run the provided training command on a system with similar specifications.
2. Monitor memory usage using htop or free -h while the script is running.
3. Check kernel logs (journalctl | grep "Out of memory") after the process is killed.

Potential Root Cause

• The script may have a memory leak or excessively large memory allocations that exceed physical RAM.
• The kernel's swap utilization logic may not engage swap space efficiently under high memory pressure.

Request for Help

I would appreciate any guidance on:

1. Debugging and resolving potential memory leaks in the script.
2. Optimizing memory usage for large-scale training processes.
3. Configuring the system (e.g., kernel, swap, ZRAM) to handle high memory usage scenarios better.

Additional Notes

Please let me know if you need additional logs, configurations, or test results to help identify the issue.

[torzdf]

There isn't enough information about your setup here.

This:

python run.py train -A /home/deepfake/workspace/faceA/... \

is not the supported way to run faceswap, which suggest a non-standard install

If you require assistance you'll need to provide the following:
https://forum.faceswap.dev/app.php/rules?sid=17e3bd94e6308c969278ddb8f25f8a6e#rule-4b

[linhcentrio]

11/17/2024 04:28:46 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'zoom_amount', datatype: '<class 'int'>', default: '5', info: 'Percentage amount to randomly zoom each training image in and out.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
11/17/2024 04:28:46 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'rotation_range', datatype: '<class 'int'>', default: '10', info: 'Percentage amount to randomly rotate each training image.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
11/17/2024 04:28:46 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'shift_range', datatype: '<class 'int'>', default: '5', info: 'Percentage amount to randomly shift each training image horizontally and vertically.', rounding: '1', min_max: (0, 25), choices: None, gui_radio: False, fixed: True, group: image augmentation)
11/17/2024 04:28:46 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'flip_chance', datatype: '<class 'int'>', default: '50', info: 'Percentage chance to randomly flip each training image horizontally.\nNB: This is ignored if the 'no-flip' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: True, group: image augmentation)
11/17/2024 04:28:46 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_lightness', datatype: '<class 'int'>', default: '30', info: 'Percentage amount to randomly alter the lightness of each training image.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: True, group: color augmentation)
11/17/2024 04:28:46 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_ab', datatype: '<class 'int'>', default: '8', info: 'Percentage amount to randomly alter the 'a' and 'b' colors of the L*a*b* color space of each training image.\nNB: This is ignored if the 'no-augment-color' optionis enabled', rounding: '1', min_max: (0, 50), choices: None, gui_radio: False, fixed: True, group: color augmentation)
11/17/2024 04:28:46 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_clahe_chance', datatype: '<class 'int'>', default: '50', info: 'Percentage chance to perform Contrast Limited Adaptive Histogram Equalization on each training image.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (0, 75), choices: None, gui_radio: False, fixed: False, group: color augmentation)
11/17/2024 04:28:46 MainProcess     _training                      config          add_item                       DEBUG    Add item: (section: 'trainer.original', title: 'color_clahe_max_size', datatype: '<class 'int'>', default: '4', info: 'The grid size dictates how much Contrast Limited Adaptive Histogram Equalization is performed on any training image selected for clahe. Contrast will be applied randomly with a gridsize of 0 up to the maximum. This value is a multiplier calculated from the training image size.\nNB: This is ignored if the 'no-augment-color' option is enabled', rounding: '1', min_max: (1, 8), choices: None, gui_radio: False, fixed: True, group: color augmentation)
11/17/2024 04:28:46 MainProcess     _training                      config          _load_defaults_from_module     DEBUG    Added defaults: trainer.original
11/17/2024 04:28:46 MainProcess     _training                      config          _handle_config                 DEBUG    Handling config: (section: model.phaze_a, configfile: '/home/deepfake/deepfake/config/train.ini')
11/17/2024 04:28:46 MainProcess     _training                      config          _check_exists                  DEBUG    Config file exists: '/home/deepfake/deepfake/config/train.ini'
11/17/2024 04:28:46 MainProcess     _training                      config          _load_config                   VERBOSE  Loading config: '/home/deepfake/deepfake/config/train.ini'
11/17/2024 04:28:46 MainProcess     _training                      config          _validate_config               DEBUG    Validating config
11/17/2024 04:28:46 MainProcess     _training                      config          _check_config_change           DEBUG    Default config has not changed
11/17/2024 04:28:46 MainProcess     _training                      config          _check_config_choices          DEBUG    Checking config choices
11/17/2024 04:28:46 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'keras_encoder' to list ['keras_encoder'] for section 'model.phaze_a', option 'freeze_layers'
11/17/2024 04:28:46 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'encoder' to list ['encoder'] for section 'model.phaze_a', option 'load_layers'
11/17/2024 04:28:46 MainProcess     _training                      config          _check_config_choices          DEBUG    Checked config choices
11/17/2024 04:28:46 MainProcess     _training                      config          _validate_config               DEBUG    Validated config
11/17/2024 04:28:46 MainProcess     _training                      config          _handle_config                 DEBUG    Handled config
11/17/2024 04:28:46 MainProcess     _training                      config          __init__                       DEBUG    Initialized: Config
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'learning_rate')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 5e-05)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'epsilon_exponent')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: -7)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'save_optimizer')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: exit)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'autoclip')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'allow_growth')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'mixed_precision')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: False)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'nan_protection')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'bool'>, value: True)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global', option: 'convert_batchsize')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 16)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: ssim)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_2')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mse)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_2')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 100)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_3')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: None)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_3')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 0)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_function_4')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: None)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'loss_weight_4')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 0)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mask_loss_function')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'str'>, value: mse)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'eye_multiplier')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 3)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mouth_multiplier')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 2)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mask_dilation')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 0.0)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mask_blur_kernel')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 3)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'global.loss', option: 'mask_threshold')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'int'>, value: 4)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'model.phaze_a', option: 'fc_dropout')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 0.0)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'model.phaze_a', option: 'fc_gblock_dropout')
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'float'>, value: 0.0)
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Getting config item: (section: 'model.phaze_a', option: 'freeze_layers')
11/17/2024 04:28:46 MainProcess     _training                      config          _parse_list                    DEBUG    Processed raw option 'keras_encoder' to list ['keras_encoder'] for section 'model.phaze_a', option 'freeze_layers'
11/17/2024 04:28:46 MainProcess     _training                      config          get                            DEBUG    Returning item: (type: <class 'list'>, value: ['keras_encoder'])
11/17/2024 04:28:46 MainProcess     _training                      config          changeable_items               DEBUG    Alterable for existing models: {'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'mask_dilation': 0.0, 'mask_blur_kernel': 3, 'mask_threshold': 4, 'fc_dropout': 0.0, 'fc_gblock_dropout': 0.0, 'freeze_layers': ['keras_encoder']}
11/17/2024 04:28:46 MainProcess     _training                      model           __init__                       DEBUG    Initializing State: (model_dir: '/home/deepfake/workspace/models/FINAL-TIENSY-FACEA_720p-FINAL-TIENSY-FACEB_1080p', model_name: 'phaze_a', config_changeable_items: '{'learning_rate': 5e-05, 'epsilon_exponent': -7, 'save_optimizer': 'exit', 'autoclip': False, 'allow_growth': False, 'mixed_precision': False, 'nan_protection': True, 'convert_batchsize': 16, 'loss_function': 'ssim', 'loss_function_2': 'mse', 'loss_weight_2': 100, 'loss_function_3': None, 'loss_weight_3': 0, 'loss_function_4': None, 'loss_weight_4': 0, 'mask_loss_function': 'mse', 'eye_multiplier': 3, 'mouth_multiplier': 2, 'mask_dilation': 0.0, 'mask_blur_kernel': 3, 'mask_threshold': 4, 'fc_dropout': 0.0, 'fc_gblock_dropout': 0.0, 'freeze_layers': ['keras_encoder']}', no_logs: False
11/17/2024 04:28:46 MainProcess     _training                      serializer      get_serializer                 DEBUG    <lib.serializer._JSONSerializer object at 0x7f3578846f80>
11/17/2024 04:28:46 MainProcess     _training                      model           _load                          DEBUG    Loading State
11/17/2024 04:28:46 MainProcess     _training                      model           _load                          INFO     No existing state file found. Generating.
11/17/2024 04:28:46 MainProcess     _training                      model           _new_session_id                DEBUG    1
11/17/2024 04:28:46 MainProcess     _training                      model           _create_new_session            DEBUG    Creating new session. id: 1
11/17/2024 04:28:46 MainProcess     _training                      model           __init__                       DEBUG    Initialized State:
11/17/2024 04:28:46 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Settings: (arguments: Namespace(func=<bound method ScriptExecutor.execute_script of <lib.cli.launcher.ScriptExecutor object at 0x7f3627c9a7d0>>, exclude_gpus=None, configfile=None, loglevel='INFO', logfile=None, redirect_gui=False, depr_logfile_LF_F=None, input_a='/home/deepfake/workspace/faceA/FINAL-TIENSY-FACEA_720p', input_b='/home/deepfake/workspace/faceB/FINAL-TIENSY-FACEB_1080p', model_dir='/home/deepfake/workspace/models/FINAL-TIENSY-FACEA_720p-FINAL-TIENSY-FACEB_1080p', load_weights=None, trainer='phaze-a', summary=False, freeze_weights=False, batch_size=16, iterations=1000000, distribution_strategy='default', no_logs=False, use_lr_finder=True, save_interval=360, snapshot_interval=10000, timelapse_input_a='/home/deepfake/workspace/faceA/FINAL-TIENSY-FACEA_720p', timelapse_input_b='/home/deepfake/workspace/faceB/FINAL-TIENSY-FACEB_1080p', timelapse_output='/home/deepfake/workspace/timelapse/FINAL-TIENSY-FACEA_720p-FINAL-TIENSY-FACEB_1080p', preview=False, write_image=False, warp_to_landmarks=True, no_flip=False, no_augment_color=False, no_warp=False, depr_summary_su_u=False, depr_iterations_it_i=None, **{'depr_batch-size_bs_b': None, 'depr_no-logs_nl_n': False, 'depr_snapshot-interval_ss_I': None, 'depr_timelapse-input-A_tia_x': None, 'depr_timelapse-input-B_tib_y': None, 'depr_timelapse-output_to_z': None, 'depr_warp-to-landmarks_wl_M': False, 'depr_no-flip_nf_P': False, 'depr_no-augment-color_nac_c': False, 'depr_no-warp_nw_W': False}), mixed_precision: False, allow_growth: False, is_predict: False)
11/17/2024 04:28:46 MainProcess     _training                      settings        _set_tf_settings               DEBUG    Not setting any specific Tensorflow settings
11/17/2024 04:28:46 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    use_mixed_precision: False
11/17/2024 04:28:46 MainProcess     _training                      settings        _set_keras_mixed_precision     DEBUG    Disabling mixed precision. (Compute dtype: float32, variable_dtype: float32)
11/17/2024 04:28:46 MainProcess     _training                      settings        _get_strategy                  DEBUG    Using strategy: <tensorflow.python.distribute.distribute_lib._DefaultDistributionStrategy object at 0x7f35788457b0>
11/17/2024 04:28:46 MainProcess     _training                      settings        __init__                       DEBUG    Initialized Settings
11/17/2024 04:28:46 MainProcess     _training                      settings        __init__                       DEBUG    Initializing Loss: (color_order: bgr)
11/17/2024 04:28:46 MainProcess     _training                      settings        _get_mask_channels             DEBUG    uses_masks: (True, True, True), mask_channels: [3, 4, 5]
11/17/2024 04:28:46 MainProcess     _training                      settings        __init__                       DEBUG    Initialized: Loss
11/17/2024 04:28:46 MainProcess     _training                      model           __init__                       DEBUG    Initialized ModelBase (Model)
11/17/2024 04:28:46 MainProcess     _training                      phaze_a         _select_freeze_layers          DEBUG    Removing 'keras_encoder' for 'fs_original'
11/17/2024 04:28:46 MainProcess     _training                      phaze_a         _get_input_shape               DEBUG    Encoder input set to: (64, 64, 3)
11/17/2024 04:28:46 MainProcess     _training                      settings        strategy_scope                 DEBUG    Using strategy scope: <tensorflow.python.distribute.distribute_lib._DefaultDistributionContext object at 0x7f3579260300>
11/17/2024 04:28:46 MainProcess     _training                      io              load                           DEBUG    Loading model: /home/deepfake/workspace/models/FINAL-TIENSY-FACEA_720p-FINAL-TIENSY-FACEB_1080p/phaze_a.h5
11/17/2024 04:28:46 MainProcess     _training                      multithreading  run                            DEBUG    Error in thread (_training): Unable to synchronously open file (truncated file: eof = 96, sblock->base_addr = 0, stored_eof = 2048)
11/17/2024 04:28:46 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
11/17/2024 04:28:46 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
11/17/2024 04:28:46 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
11/17/2024 04:28:46 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
11/17/2024 04:28:46 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
11/17/2024 04:28:46 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training'
11/17/2024 04:28:46 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training'
Traceback (most recent call last):
  File "/home/deepfake/deepfake/lib/cli/launcher.py", line 225, in execute_script
    process.process()
  File "/home/deepfake/deepfake/scripts/train.py", line 204, in process
    self._end_thread(thread, err)
  File "/home/deepfake/deepfake/scripts/train.py", line 244, in _end_thread
    thread.join()
  File "/home/deepfake/deepfake/lib/multithreading.py", line 224, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/home/deepfake/deepfake/lib/multithreading.py", line 100, in run
    self._target(*self._args, **self._kwargs)
  File "/home/deepfake/deepfake/scripts/train.py", line 269, in _training
    raise err
  File "/home/deepfake/deepfake/scripts/train.py", line 254, in _training
    model = self._load_model()
  File "/home/deepfake/deepfake/scripts/train.py", line 285, in _load_model
    model.build()
  File "/home/deepfake/deepfake/plugins/train/model/phaze_a.py", line 189, in build
    model = self.io.load()
  File "/home/deepfake/deepfake/plugins/train/model/_base/io.py", line 147, in load
    model = kmodels.load_model(self.filename, compile=False)
  File "/home/deepfake/deepfake/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/deepfake/deepfake/venv/lib/python3.10/site-packages/h5py/_hl/files.py", line 561, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/deepfake/deepfake/venv/lib/python3.10/site-packages/h5py/_hl/files.py", line 235, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 102, in h5py.h5f.open
OSError: Unable to synchronously open file (truncated file: eof = 96, sblock->base_addr = 0, stored_eof = 2048)

============ System Information ============
backend:             nvidia
encoding:            UTF-8
git_branch:          Not Found
git_commits:         Not Found
gpu_cuda:            11.8
gpu_cudnn:           8
gpu_devices:         GPU_0: NVIDIA RTX A4000
gpu_devices_active:  GPU_0
gpu_driver:          560.35.03
gpu_vram:            GPU_0: 16376MB (16003MB free)
os_machine:          x86_64
os_platform:         Linux-5.14.0-427.37.1.el9_4.x86_64-x86_64-with-glibc2.34
os_release:          5.14.0-427.37.1.el9_4.x86_64
py_command:          run.py train -A /home/deepfake/workspace/faceA/FINAL-TIENSY-FACEA_720p/ -B /home/deepfake/workspace/faceB/FINAL-TIENSY-FACEB_1080p/ -m /home/deepfake/workspace/models/FINAL-TIENSY-FACEA_720p-FINAL-TIENSY-FACEB_1080p/ -t phaze-a -b 16 -i 1000000 -D default -r -s 360 -I 10000 -x /home/deepfake/workspace/faceA/FINAL-TIENSY-FACEA_720p/ -y /home/deepfake/workspace/faceB/FINAL-TIENSY-FACEB_1080p/ -z /home/deepfake/workspace/timelapse/FINAL-TIENSY-FACEA_720p-FINAL-TIENSY-FACEB_1080p/ -M -L INFO
py_conda_version:    conda 24.7.1
py_implementation:   CPython
py_version:          3.10.15
py_virtual_env:      False
sys_cores:           24
sys_processor:       x86_64
sys_ram:             Total: 31836MB, Available: 30616MB, Used: 812MB, Free: 28534MB

=============== Pip Packages ===============
absl-py==2.1.0
aiofiles==23.2.1
annotated-types==0.7.0
anyio==4.6.2.post1
astunparse==1.6.3
cachetools==5.5.0
certifi==2022.12.7
charset-normalizer==2.1.1
click==8.1.7
contourpy==1.3.1
cycler==0.12.1
exceptiongroup==1.2.2
fastapi==0.115.5
fastcluster==1.2.6
ffmpy==0.4.0
filelock==3.13.1
flatbuffers==24.3.25
fonttools==4.55.0
fsspec==2024.2.0
gast==0.4.0
google-auth==2.36.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
gradio==5.6.0
gradio_client==1.4.3
grpcio==1.68.0
h11==0.14.0
h5py==3.12.1
httpcore==1.0.7
httpx==0.27.2
huggingface-hub==0.26.2
idna==3.4
imageio==2.36.0
imageio-ffmpeg==0.5.1
Jinja2==3.1.3
joblib==1.4.2
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.7
libclang==18.1.1
Markdown==3.7
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.9.2
mdurl==0.1.2
mpmath==1.3.0
networkx==3.2.1
numexpr==2.10.1
numpy==1.26.3
nvidia-ml-py==12.560.30
oauthlib==3.2.2
opencv-python==4.10.0.84
opt_einsum==3.4.0
orjson==3.10.11
packaging==24.2
pandas==2.2.3
Pillow==9.5.0
protobuf==3.19.6
psutil==6.1.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
pydantic==2.9.2
pydantic_core==2.23.4
pydub==0.25.1
Pygments==2.18.0
pyparsing==3.2.0
python-dateutil==2.9.0.post0
python-multipart==0.0.12
pytz==2024.2
PyYAML==6.0.2
requests==2.28.1
requests-oauthlib==2.0.0
rich==13.9.4
rsa==4.9
ruff==0.7.4
safehttpx==0.1.1
scikit-learn==1.5.2
scipy==1.14.1
semantic-version==2.10.0
shellingham==1.5.4
six==1.16.0
sniffio==1.3.1
starlette==0.41.2
sympy==1.13.1
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.1
tensorflow-estimator==2.10.0
tensorflow-io-gcs-filesystem==0.37.1
termcolor==2.5.0
threadpoolctl==3.5.0
tomlkit==0.12.0
torch==2.1.0+cu118
torchaudio==2.1.0+cu118
torchvision==0.16.0+cu118
tqdm==4.67.0
triton==2.1.0
typer==0.13.0
typing_extensions==4.9.0
tzdata==2024.2
urllib3==1.26.13
uvicorn==0.32.0
websockets==12.0
Werkzeug==3.1.3
wrapt==1.16.0
xformers==0.0.22.post7+cu118

============== Conda Packages ==============
# packages in environment at /home/deepfake/deepfake/venv:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   2.1.0                    pypi_0    pypi
aiofiles                  23.2.1                   pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
anyio                     4.6.2.post1              pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
bzip2                     1.0.8                h5eee18b_6  
ca-certificates           2024.9.24            h06a4308_0  
cachetools                5.5.0                    pypi_0    pypi
certifi                   2022.12.7                pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
contourpy                 1.3.1                    pypi_0    pypi
cuda-cccl                 11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-command-line-tools   11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-compiler             11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-cudart               11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-cudart-dev           11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-cuobjdump            11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-cupti                11.8.87                       0    nvidia/label/cuda-11.8.0
cuda-cuxxfilt             11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-documentation        11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-driver-dev           11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-gdb                  11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-libraries            11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-libraries-dev        11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-memcheck             11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nsight               11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nsight-compute       11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-nvcc                 11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-nvdisasm             11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nvml-dev             11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nvprof               11.8.87                       0    nvidia/label/cuda-11.8.0
cuda-nvprune              11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nvrtc                11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-nvrtc-dev            11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-nvtx                 11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nvvp                 11.8.87                       0    nvidia/label/cuda-11.8.0
cuda-profiler-api         11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-sanitizer-api        11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-toolkit              11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-tools                11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-version              12.6                 h7480c83_3    conda-forge
cuda-visual-tools         11.8.0                        0    nvidia/label/cuda-11.8.0
cudnn                     8.9.7.29             h092f7fd_3    conda-forge
cycler                    0.12.1                   pypi_0    pypi
exceptiongroup            1.2.2                    pypi_0    pypi
fastapi                   0.115.5                  pypi_0    pypi
fastcluster               1.2.6                    pypi_0    pypi
ffmpy                     0.4.0                    pypi_0    pypi
filelock                  3.13.1                   pypi_0    pypi
flatbuffers               24.3.25                  pypi_0    pypi
fonttools                 4.55.0                   pypi_0    pypi
fsspec                    2024.2.0                 pypi_0    pypi
gast                      0.4.0                    pypi_0    pypi
gds-tools                 1.4.0.31                      0    nvidia/label/cuda-11.8.0
google-auth               2.36.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
gradio                    5.6.0                    pypi_0    pypi
gradio-client             1.4.3                    pypi_0    pypi
grpcio                    1.68.0                   pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
h5py                      3.12.1                   pypi_0    pypi
httpcore                  1.0.7                    pypi_0    pypi
httpx                     0.27.2                   pypi_0    pypi
huggingface-hub           0.26.2                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
imageio                   2.36.0                   pypi_0    pypi
imageio-ffmpeg            0.5.1                    pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
joblib                    1.4.2                    pypi_0    pypi
keras                     2.10.0                   pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.7                    pypi_0    pypi
ld_impl_linux-64          2.40                 h12ee557_0  
libclang                  18.1.1                   pypi_0    pypi
libcublas                 11.11.3.6                     0    nvidia/label/cuda-11.8.0
libcublas-dev             11.11.3.6                     0    nvidia/label/cuda-11.8.0
libcufft                  10.9.0.58                     0    nvidia/label/cuda-11.8.0
libcufft-dev              10.9.0.58                     0    nvidia/label/cuda-11.8.0
libcufile                 1.4.0.31                      0    nvidia/label/cuda-11.8.0
libcufile-dev             1.4.0.31                      0    nvidia/label/cuda-11.8.0
libcurand                 10.3.0.86                     0    nvidia/label/cuda-11.8.0
libcurand-dev             10.3.0.86                     0    nvidia/label/cuda-11.8.0
libcusolver               11.4.1.48                     0    nvidia/label/cuda-11.8.0
libcusolver-dev           11.4.1.48                     0    nvidia/label/cuda-11.8.0
libcusparse               11.7.5.86                     0    nvidia/label/cuda-11.8.0
libcusparse-dev           11.7.5.86                     0    nvidia/label/cuda-11.8.0
libffi                    3.4.4                h6a678d5_1  
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
libnpp                    11.8.0.86                     0    nvidia/label/cuda-11.8.0
libnpp-dev                11.8.0.86                     0    nvidia/label/cuda-11.8.0
libnvjpeg                 11.9.0.86                     0    nvidia/label/cuda-11.8.0
libnvjpeg-dev             11.9.0.86                     0    nvidia/label/cuda-11.8.0
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libuuid                   1.41.5               h5eee18b_0  
libzlib                   1.2.13               h4ab18f5_6    conda-forge
markdown                  3.7                      pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.9.2                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.2.1                    pypi_0    pypi
nsight-compute            2022.3.0.22                   0    nvidia/label/cuda-11.8.0
numexpr                   2.10.1                   pypi_0    pypi
numpy                     1.26.3                   pypi_0    pypi
nvidia-ml-py              12.560.30                pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python             4.10.0.84                pypi_0    pypi
openssl                   3.4.0                hb9d3cd8_0    conda-forge
opt-einsum                3.4.0                    pypi_0    pypi
orjson                    3.10.11                  pypi_0    pypi
packaging                 24.2                     pypi_0    pypi
pandas                    2.2.3                    pypi_0    pypi
pillow                    9.5.0                    pypi_0    pypi
pip                       24.2            py310h06a4308_0  
protobuf                  3.19.6                   pypi_0    pypi
psutil                    6.1.0                    pypi_0    pypi
pyasn1                    0.6.1                    pypi_0    pypi
pyasn1-modules            0.4.1                    pypi_0    pypi
pydantic                  2.9.2                    pypi_0    pypi
pydantic-core             2.23.4                   pypi_0    pypi
pydub                     0.25.1                   pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
pyparsing                 3.2.0                    pypi_0    pypi
python                    3.10.15              he870216_1  
python-dateutil           2.9.0.post0              pypi_0    pypi
python-multipart          0.0.12                   pypi_0    pypi
pytz                      2024.2                   pypi_0    pypi
pyyaml                    6.0.2                    pypi_0    pypi
readline                  8.2                  h5eee18b_0  
requests                  2.28.1                   pypi_0    pypi
requests-oauthlib         2.0.0                    pypi_0    pypi
rich                      13.9.4                   pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
ruff                      0.7.4                    pypi_0    pypi
safehttpx                 0.1.1                    pypi_0    pypi
scikit-learn              1.5.2                    pypi_0    pypi
scipy                     1.14.1                   pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
setuptools                75.1.0          py310h06a4308_0  
shellingham               1.5.4                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
sqlite                    3.45.3               h5eee18b_0  
starlette                 0.41.2                   pypi_0    pypi
sympy                     1.13.1                   pypi_0    pypi
tensorboard               2.10.1                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.10.1                   pypi_0    pypi
tensorflow-estimator      2.10.0                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.37.1                   pypi_0    pypi
termcolor                 2.5.0                    pypi_0    pypi
threadpoolctl             3.5.0                    pypi_0    pypi
tk                        8.6.14               h39e8969_0  
tomlkit                   0.12.0                   pypi_0    pypi
torch                     2.1.0+cu118              pypi_0    pypi
torchaudio                2.1.0+cu118              pypi_0    pypi
torchvision               0.16.0+cu118             pypi_0    pypi
tqdm                      4.67.0                   pypi_0    pypi
triton                    2.1.0                    pypi_0    pypi
typer                     0.13.0                   pypi_0    pypi
typing-extensions         4.9.0                    pypi_0    pypi
tzdata                    2024.2                   pypi_0    pypi
urllib3                   1.26.13                  pypi_0    pypi
uvicorn                   0.32.0                   pypi_0    pypi
websockets                12.0                     pypi_0    pypi
werkzeug                  3.1.3                    pypi_0    pypi
wheel                     0.44.0          py310h06a4308_0  
wrapt                     1.16.0                   pypi_0    pypi
xformers                  0.0.22.post7+cu118          pypi_0    pypi
xz                        5.4.6                h5eee18b_1  
zlib                      1.2.13               h4ab18f5_6    conda-forge

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0
erosion_top:              0.0
erosion_bottom:           0.0
erosion_left:             0.0
erosion_right:            0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
separate_mask:            False
jpg_quality:              75
png_compress_level:       3

[writer.patch]
start_index:              0
index_offset:             0
number_padding:           6
include_filename:         True
face_index_location:      before
origin:                   bottom-left
empty_frames:             blank
json_output:              False
separate_mask:            False
bit_depth:                16
format:                   png
png_compress_level:       3
tiff_compression_method:  lzw

[writer.pillow]
format:                   png
draw_transparent:         False
separate_mask:            False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False
aligner_min_scale:        0.07
aligner_max_scale:        2.0
aligner_distance:         22.5
aligner_roll:             45.0
aligner_features:         True
filter_refeed:            True
save_filtered:            False
realign_refeeds:          True
filter_realign:           True

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
cpu:                      True
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.bisenet_fp]
batch-size:               8
cpu:                      False
weights:                  faceswap
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.custom]
batch-size:               8
centering:                face
fill:                     False

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

[recognition.vgg_face2]
batch-size:               16
cpu:                      False

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                face
coverage:                 87.5
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
save_optimizer:           exit
lr_finder_iterations:     1000
lr_finder_mode:           set
lr_finder_strength:       default
autoclip:                 False
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
loss_function_2:          mse
loss_weight_2:            100
loss_function_3:          none
loss_weight_3:            0
loss_function_4:          none
loss_weight_4:            0
mask_loss_function:       mse
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_dilation:            0
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.phaze_a]
output_size:              128
shared_fc:                none
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              7
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          none
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_upscales_in_fc:       0
dec_norm:                 none
dec_min_filters:          64
dec_max_filters:          512
dec_slope_mode:           full
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
fs_original_use_alt:      False
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
mask_opacity:             30
mask_color:               #ff0000
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

[torzdf]

This:

OSError: Unable to synchronously open file (truncated file: eof = 96, sblock->base_addr = 0, stored_eof = 2048)

Is a corrupted model file.

Where did you install faceswap from? You have non-standard packages in your environment, and you use a non standard command for launching faceswap