Timeout during deploy:compile_assets

[DalfDev]

Configuration

https://gist.github.com/DalfDev/eddaacb4f625c7f63f9c8e20f5dc26c9

Expected behavior

It should compile assets and deploy after and then terminate compilation processes.

Actual behavior

Most of the compilation of our assets timeout. Plus, the processes are staying on our server.

On this screenshot (taken on 2020-09-10), you can see two compilations from the day before (2020-09-09):
https://image.noelshack.com/fichiers/2020/37/5/1599831999-screenshot-2020-09-10-at-09-03-55.png

Here an example of a deploy:compile_assets we had to interrupt after a while:
https://image.noelshack.com/fichiers/2020/37/5/1599831998-screenshot-2020-09-11-at-15-27-57.png

This seems to happen when processes are not killed. After a reboot, everything works fine the first time but killing the remaining processes manually is not enough to fix the problem.

This problem happened since we upgraded our rails version from 5.2.0 to 6.0.3.

System configuration

https://gist.github.com/DalfDev/153324537f86924edf504f443e0100b5

[mattbrictson]

Compiling assets is known to be a pretty memory-intensive operation. It is possible that your server is running out of memory and it starts to swap, thereby grinding to a halt. The fact that you see different behavior on a fresh boot supports this theory, because the server will have more free memory after a reboot.

FYI, Capistrano works via SSH and interrupting a command that is run over SSH does not interrupt the command on the server. In other words, if Capistrano kicks off a long-running command on the server and then you ctrl-C to kill Capistrano, the command on the server will keep going.

I don't have a good suggestion for you other than try running top on the server while all this is happening and see what the memory situation looks like. If it is indeed the case that you are running low on free memory or if swap is being used, you have two options: (1) add more memory to the server or switch to a different server that has higher specs, (2) figure out a different deployment solution that doesn't involve precompiling. For example I think some people compile locally and then upload the assets to S3 so that they don't have to compile them on the server. I haven't done that myself so I can't really give much more direction than that.

Finally, when you compile assets for a Rails project that typically means the actual compilation is being done by Sprockets, Webpacker, or both. I'm going to guess that you are using both because the output shows yarn install being run, which would suggest Webpacker, but you are using public/assets in your :linked_dirs, which suggests Sprockets. Another avenue to take is to look at the GitHub issues for those projects and see if other people are having similar problems and have found solutions. I suspect that when you upgraded Rails you probably also updated one or both of those dependencies as well, which may explain the change in behavior.

[eoinkelly]

This can also be because the SSH server decides the connection is dead because webpack doesn't emit any output during it's Compiling... phase.

I have encountered this and fixed it by adding the following to the sshd config on the server (probably in /etc/ssh/sshd_config but that might vary by distro)

  # ClientAliveInterval is important for apps with slow assets:precompile
  # because it stops the server from disconnecting Capistrano if there hasn't
  # been any output for a while
  ClientAliveInterval 30