multiple daphne processes with supervisord and UNIX socket #287

JanMalte · 2019-11-12T13:49:05Z

Setup daphne with supervisord and nginx as shown in the docs is working.
https://channels.readthedocs.io/en/latest/deploying.html#nginx-supervisor-ubuntu

But changing from TCP to UNIX socket for the fcgi process group results in daphne not responding to requests.

Run as single program in supervisor

[program:current.example.com_daphne]
command = /srv/current.example.com/env/bin/daphne --unix-socket /srv/current.example.com/run/daphne.sock --proxy-headers crm.asgi:application
directory = /srv/current.example.com/src/
environment = LANGUAGE=en_US.UTF-8,LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8,PYTHONPATH='/srv/current.example.com/env/'
user = www-data

Request

root@crm:# curl -v --no-buffer --unix-socket /srv/current.example.com/run/daphne.sock "http://current.example.com/dashboard/"
*   Trying /srv/current.example.com/run/daphne.sock...
* Connected to current.example.com (/srv/current.example.com/run/daphne.sock) port 80 (#0)
> GET /dashboard/ HTTP/1.1
> Host: current.example.com
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Content-Type: text/html; charset=utf-8
< Location: /auth/login/?next=/dashboard/
< X-Frame-Options: SAMEORIGIN
< Vary: Accept-Language, Cookie
< Content-Language: de
< Content-Length: 0
< 
* Connection #0 to host current.example.com left intact

Run as fcgi-program in supervisor

[fcgi-program:demo.example.com_daphne]
command = /srv/demo.example.com/env/bin/daphne --fd 0 --proxy-headers crm.asgi:application
directory = /srv/demo.example.com/src/
environment = LANGUAGE=en_US.UTF-8,LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8,PYTHONPATH='/srv/demo.example.com/env/'
user = www-data
socket=unix:///srv/demo.example.com/run/daphne.sock
numprocs = 2

Request

root@crm:# curl -v --no-buffer --unix-socket /srv/demo.example.com/run/daphne.sock "http://demo.example.com/dashboard/"
*   Trying /srv/demo.example.com/run/daphne.sock...
* Connected to demo.example.com (/srv/demo.example.com/run/daphne.sock) port 80 (#0)
> GET /dashboard/ HTTP/1.1
> Host: demo.example.com
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* stopped the pause stream!
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

The text was updated successfully, but these errors were encountered:

carltongibson · 2019-11-12T15:12:44Z

Hi. There's not much to go on here, and I don't have an instant answer I'm afraid. Are you able to dig in a little to see what might be going on?

Pyvonix · 2020-03-24T12:05:09Z

I get exactly the same problem.

After differents tests, when I use the same setup:
NGINX (front as a reverse proxy) -> "a socket" -> daphne worker

with TCP socket

nginx:

upstream django {
    server 127.0.0.1:8000;
}

supervisord:

[fcgi-program:daphne]
socket = tcp://127.0.0.1:8000

numprocs = 10
process_name = %(program_name)s_%(process_num)02d

It's work great.

with UNIX socket

nginx:

upstream django {
    server unix:///tmp/unix.sock;
}

supervisord:

[fcgi-program:daphne]
socket = unix:///tmp/unix.sock
socket_owner = www-data
socket_mod = 0600

command = daphne --fd 0 --access-log - --proxy-headers app.asgi:application

numprocs = 10
process_name = %(program_name)s_%(process_num)02d

But when I passed to UNIX sock I get trouble... Nginx can't talk to my daphne server via this socket....

nginx error log:

2020/03/24 07:59:00 [error] 615#615: *31 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.2, server: ,  request: "GET / HTTP/2.0", upstream: "http://unix:///tmp/unix.sock:/", host: "192.168.0.1"

NB: everything runned by supervisord -n -c config.ini

AndreyMZ · 2020-03-27T06:33:42Z

I get exactly the same problem. It is very easy to reproduce, see the steps below. Unfortunately, I have no more time to dig the root cause.

Steps to reproduce

Extract test-daphne-287.zip.

In a first terminal run:

 cd test-daphne-287
 docker build --tag=test .
 docker run -p 127.0.0.1:8081:8081 -p 127.0.0.1:8082:8082 -it test

In a second terminal run:

 curl http://localhost:8081/
 curl http://localhost:8082/

Actual result

In the second terminal:

C:\>curl http://localhost:8081/
Hello, world!

C:\>curl http://localhost:8082/
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

In the first terminal:

172.17.0.1 - - [27/Mar/2020:05:42:13 +0000] "GET / HTTP/1.1" 200 24 "-" "curl/7.55.1"
2020/03/27 05:42:15 [error] 11#11: *3 recv() failed (104: Connection reset by peer) whle reading response header from upstream, client: 172.17.0.1, server: edebb61d789c, request: "GET / HTTP/1.1", upstream: "http://unix:/run/asgi.sock:/", host: "localhost:8082"
172.17.0.1 - - [27/Mar/2020:05:42:15 +0000] "GET / HTTP/1.1" 502 173 "-" "curl/7.55.1"

Expected result

In the second terminal:

C:\>curl http://localhost:8081/
Hello, world!

C:\>curl http://localhost:8082/
Hello, world!

In the first terminal:

172.17.0.1 - - [27/Mar/2020:05:42:13 +0000] "GET / HTTP/1.1" 200 24 "-" "curl/7.55.1"
172.17.0.1 - - [27/Mar/2020:05:42:15 +0000] "GET / HTTP/1.1" 200 24 "-" "curl/7.55.1"

BTW, uvicorn works fine. To check it just repeat the steps, but before building the Docker image replace daphne with uvicorn everywhere:

sed "s/daphne/uvicorn/" --in-place=.bak Dockerfile supervisord.conf

TheKalpit · 2020-05-19T04:19:23Z

I'm also facing this issue. supervisord + daphne + apache.
Works perfectly on TCP socket (on any port), but not on unix socket.

rrauenza · 2020-06-24T00:04:16Z

FYI, also seeing this at channels==2.4.0, daphne==2.5.0. (I'll just switch to TCP instead...)

stephendwolff · 2020-07-08T16:40:17Z

I've tried TCP and unix sockets, and i can get normal http being proxied to daphne using nginx (with an ssl layer), but websocket connections disconnect after the initial handshake. (ie i can see a normal django site views).

@JanMalte's original issue talked of the supervisord [fcgi-program] not working. The example given isn't the same as the Channels deployment doc linked (it may have changed):

Setup daphne with supervisord and nginx as shown in the docs is working.
https://channels.readthedocs.io/en/latest/deploying.html#nginx-supervisor-ubuntu

But changing from TCP to UNIX socket for the fcgi process group results in daphne not responding to requests.
Run as fcgi-program in supervisor

[fcgi-program:demo.example.com_daphne]
command = /srv/demo.example.com/env/bin/daphne --fd 0 --proxy-headers crm.asgi:application
directory = /srv/demo.example.com/src/
environment = LANGUAGE=en_US.UTF-8,LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8,PYTHONPATH='/srv/demo.example.com/env/'
user = www-data
socket=unix:///srv/demo.example.com/run/daphne.sock
numprocs = 2

Request

root@crm:# curl -v --no-buffer --unix-socket /srv/demo.example.com/run/daphne.sock "http://demo.example.com/dashboard/"
*   Trying /srv/demo.example.com/run/daphne.sock...
* Connected to demo.example.com (/srv/demo.example.com/run/daphne.sock) port 80 (#0)
> GET /dashboard/ HTTP/1.1
> Host: demo.example.com
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* stopped the pause stream!
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

https://channels.readthedocs.io/en/latest/deploying.html#nginx-supervisor-ubuntu has

command=daphne -u /run/daphne/daphne%(process_num)d.sock --fd 0 --access-log - --proxy-headers mysite.asgi:application

ie the $(process_num)d.sock. The original example didn't - i think that's why it was failing.

The other commenters appear to be missing the same variable for making a difference between process (names).

Still - doesn't help my websockets being dropped. Thought i'd chip this in, in case it helps anyone.

stephendwolff · 2020-07-09T11:25:08Z

I finally fixed my websocket connection. I discovered Ubuntu18.04 standard apt repositories provide redis v4, and Channels Redis requires >=v5.

carltongibson · 2020-07-09T11:31:42Z

Hi @stephendwolff - Glad you solved it. For note, Redis only supports the current and previous versions (so currently 6 and 5). Anything older than that is EOL. So it’s important to always keep up.

(Will add something to that effect to the docs.)

stephendwolff · 2020-07-09T11:35:53Z

@carltongibson thanks - it is in the Channels Redis docs, but i hadn't thought to look there until i got logging fully working and found where thing broke! The Redis docs recommend installing from source, which i hadn't done, as i was using Ansible for deployment and it was easier just to use apt.

Genarito · 2020-10-22T23:05:44Z

Similar to what @stephendwolff proposed, I've solved the problem updating my Redis server and dependencies. My final versions are:

Python: 3.8.5 or 3.8.6 (both tested)
Redis 6.0.8
Dependencies:

channels==2.4.0
channels-redis==3.1.0
daphne==2.5.0
Django==3.1.2
django-filter==2.4.0
djangorestframework==3.12.1

sermoz · 2023-09-28T22:12:26Z

@JanMalte were you able to find a solution to this problem? Same question for @TheKalpit and @rrauenza. The original problem remains unsolved as far as I get it.

JanMalte · 2023-09-28T22:41:19Z

Not as far as I can remember.
I think we still use TCP sockets.

carltongibson · 2024-10-08T14:16:04Z

OK, this really doesn't look like a Daphne issue per se. As face value, it would be a lower level twisted issue (perhaps) but folks seems to be resolving with supervisor config changes or updating.

I'm going to close as stale/resolved. If you hit this issue again, please open a fresh thread, linking back here, with a reproduce.

InvalidInterrupt mentioned this issue Sep 3, 2023

Detect address family when using --fd option #479

Open

carltongibson closed this as not planned Won't fix, can't repro, duplicate, stale Oct 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multiple daphne processes with supervisord and UNIX socket #287

multiple daphne processes with supervisord and UNIX socket #287

JanMalte commented Nov 12, 2019

carltongibson commented Nov 12, 2019

Pyvonix commented Mar 24, 2020 •

edited

Loading

AndreyMZ commented Mar 27, 2020

TheKalpit commented May 19, 2020

rrauenza commented Jun 24, 2020

stephendwolff commented Jul 8, 2020

stephendwolff commented Jul 9, 2020

carltongibson commented Jul 9, 2020

stephendwolff commented Jul 9, 2020

Genarito commented Oct 22, 2020

sermoz commented Sep 28, 2023

JanMalte commented Sep 28, 2023

carltongibson commented Oct 8, 2024

multiple daphne processes with supervisord and UNIX socket #287

multiple daphne processes with supervisord and UNIX socket #287

Comments

JanMalte commented Nov 12, 2019

carltongibson commented Nov 12, 2019

Pyvonix commented Mar 24, 2020 • edited Loading

AndreyMZ commented Mar 27, 2020

Steps to reproduce

Actual result

Expected result

TheKalpit commented May 19, 2020

rrauenza commented Jun 24, 2020

stephendwolff commented Jul 8, 2020

stephendwolff commented Jul 9, 2020

carltongibson commented Jul 9, 2020

stephendwolff commented Jul 9, 2020

Genarito commented Oct 22, 2020

sermoz commented Sep 28, 2023

JanMalte commented Sep 28, 2023

carltongibson commented Oct 8, 2024

Pyvonix commented Mar 24, 2020 •

edited

Loading