bug: OT-2 stuck in a boot loop following software update?

[Parasiterex]

Overview

When starting the OT-2 Robot it will boot for about 1 minute, and then reset itself.

During this time there is a brief connection to the OT-2 app which then states that the robot is not responding correct the request from the api-server@local ip address.

Connection via ssh is possible during for about 30 seconds after boot, until it resets again.

nginx logs from pi indicate the following

2022/12/20 18:57:26 [crit] 191#0: *8 connect() to unix:/run/aiohttp.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.146.226, server: , request: "GET /networking/status HTTP/1.1", upstream: "http://unix:/run/aiohttp.sock:/networking/status", host: "169.254.90.58:31950"
2022/12/20 18:57:31 [crit] 191#0: *8 connect() to unix:/run/aiohttp.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.146.226, server: , request: "GET /networking/status HTTP/1.1", upstream: "http://unix:/run/aiohttp.sock:/networking/status", host: "169.254.90.58:31950"
2022/12/20 18:57:36 [crit] 191#0: *8 connect() to unix:/run/aiohttp.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.146.226, server: , request: "GET /networking/status HTTP/1.1", upstream: "http://unix:/run/aiohttp.sock:/networking/status", host: "169.254.90.58:31950"
2022/12/20 18:57:40 [crit] 191#0: *17 connect() to unix:/run/aiohttp.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.146.226, server: , request: "GET /health HTTP/1.1", upstream: "http://unix:/run/aiohttp.sock:/health", host: "169.254.90.58:31950"
2022/12/20 18:57:41 [crit] 191#0: *8 connect() to unix:/run/aiohttp.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.146.226, server: , request: "GET /networking/status HTTP/1.1", upstream: "
http://unix:/run/aiohttp.sock:/networking/status", host: "169.254.90.58:31950"
2022/12/20 18:57:46 [crit] 191#0: *8 connect() to unix:/run/aiohttp.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.146.226, server: , request: "GET /networking/status HTTP/1.1", upstream: "http://unix:/run/aiohttp.sock:/networking/status", host: "169.254.90.58:31950"
2022/12/20 18:57:51 [crit] 191#0: *8 connect() to unix:/run/aiohttp.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.146.226, server: , request: "GET /networking/status HTTP/1.1", upstream: "http://unix:/run/aiohttp.sock:/networking/status", host: "169.254.90.58:31950"
2022/12/20 18:57:55 [crit] 191#0: *24 connect() to unix:/run/aiohttp.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.146.226, server: , request: "GET /health HTTP/1.1", upstream: "http://unix:/run/aiohttp.sock:/health", host: "169.254.90.58:31950"
2022/12/20 18:57:56 [crit] 191#0: *8 connect() to unix:/run/aiohttp.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.146.226, server: , request: "GET /networking/status HTTP/1.1", upstream: "http://unix:/run/aiohttp.sock:/networking/status", host: "169.254.90.58:31950"

while logs from opentrons indicate the following:

{"level":"debug","message":"Stopping mDNS browser","timestamp":"2024-11-07T13:08:57.388Z","label":"discovery","meta":{}}
{"level":"debug","message":"Creating _http._tcp mDNS browser","timestamp":"2024-11-07T13:08:57.388Z","label":"discovery","meta":{"ports":[31950]}}
{"level":"debug","message":"Sending mDNS discovery query","timestamp":"2024-11-07T13:08:57.391Z","label":"discovery","meta":{}}
{"level":"debug","message":"No Flex serial port found.","timestamp":"2024-11-07T13:09:00.189Z","label":"usb","meta":{}}
{"level":"debug","message":"Sending mDNS discovery query","timestamp":"2024-11-07T13:09:01.396Z","label":"discovery","meta":{}}
{"level":"debug","message":"Received action from main via IPC","timestamp":"2024-11-07T13:09:01.409Z","label":"renderer","meta":{"actionType":"systemInfo:USB_DEVICE_ADDED"}}
{"level":"debug","message":"Received action via IPC from renderer","timestamp":"2024-11-07T13:09:01.410Z","label":"main","meta":{"action":{"type":"systemInfo:USB_DEVICE_ADDED","payload":{"usbDevice":{"vendorId":3034,"productId":33107,"identifier":"809be379355caf5c673285b4efca79a7","manufacturerName":"Realtek","productName":"Realtek USB GbE Family Controller","systemIdentifier":"USB\VID_0BDA&PID_8153\000001","windowsDriverVersion":"10.61.20.327"}},"meta":{"shell":true}}}}
{"level":"debug","message":"handling action in discovery","timestamp":"2024-11-07T13:09:01.410Z","label":"discovery","meta":{"action":{"type":"systemInfo:USB_DEVICE_ADDED","payload":{"usbDevice":{"vendorId":3034,"productId":33107,"identifier":"809be379355caf5c673285b4efca79a7","manufacturerName":"Realtek","productName":"Realtek USB GbE Family Controller","systemIdentifier":"USB\VID_0BDA&PID_8153\000001","windowsDriverVersion":"10.61.20.327"}},"meta":{"shell":true}}}}

Steps to reproduce

I believe this was following the latest 8.1.0 app update, but rollbacks, and reflashing the robot to a previous version shows the same issues.

Current behavior

Bootloop, not connecting to windows pc

Expected behavior

After booting OT-2 should connect to the system.

Operating system

Windows

System and robot setup or anything else?

App version 8.1.0, latest Robot-files. Connect via USB-over-Ethernet.

[rogeriog]

I believe I am having the same issue, I am now going to try to rollback the software and robot versions...
EDIT: Rolledback to versions 7.3.0, for both the app and robot, which I believe where the ones it came with, still same error.
I will attach logs from the robot and screenshots. Hope someone can help.
OT2CEP20220118R02_logs.zip



The error appears when I first connect to the OT-2 as the OP describes, and then it appears again whenever I try to start setup of a protocol. Also, this error regarding the robot settings and deck calibration wasn't appearing before when I analyzed the protocol was uploaded. I'm considering a factory reset now..