Outlook/best practices for custom packages

[pbeaucage]

I have a use case for Opentrons robots that's a little outside the mainstream, where I have a custom server that runs on the robot that provides ad hoc, atomic command support outside of the idea of a protocol run where state variables like tips remaining are managed on the robot. (essentially, the robot is making mixtures on the fly based on characterization of prior samples). My codebase goes back into ancient Opentrons history, at least back to software 3.0, and I realize that at this point the HTTP API would probably be the recommended way to do this, though my understanding is that HTTP also has some underlying trouble with this kind of non-protocol running.

The problem I run into is that every software update brings a new set of packages on the robot side, so every version upgrade I contemplate, I wind up having to engineer varying levels of unstable hacks to get my code to run. By way of example, on 4.7.0, I have to modify distribution Python packages to assert Py3.7 compatibility and extract binaries for some compiled Python packages from universal wheels which are not supported on pip that old, on 7.2.0 I need to compile my own PyYAML on another armv7l box with handbuilt Py3.10 and transport the binary onto the robot, need to compile my own copy of git or use a pure-Python reimplementation of it since that was silently removed, etc, etc. There are features I don't run on the robot not because of compute power but because scipy is just too difficult to install.

As this project has grown we have a growing fleet of 7 OT-2 robots at 4 sites around the US running this codebase and the effort required to push these hacks is becoming unsustainable. I'm curious if there is an official solution contemplated or coming that would allow user-partition installs of these kinds of relatively basic, but compiled, packages. If not, I'm curious if someone would be able to offer thoughts on generating a fork of Opentrons buildroot where I could just repartition the SD card to expand the root partition, then install large swaths of Python and other compiled packages that would address almost all anticipated needs. It seems like buildroot provides a lot of desired packages, but the root partition size on OT-2 prevents shipping them with the robot, which is a choice I understand.

Obviously I would need to turn off firmware update signing on the robots that would receive this custom firmware, and would be responsible for syncing my buildroot fork with this one when updates come, but it seems more sustainable than having a manual repository of binary blobs and a hacked-together setup script for every Opentrons update. Before going further down this road, I figured it would be prudent to check in with experts and see if this sounds like a reasonable path. Maybe such a "developer" firmware with a large package base that required a complete SD card reflash that expanded the root partition could even be officially distributed if OT envisions running complex code on the robots as a viable path forward.

[sfoster1]

Hey @pbeaucage, thanks for reaching out!

I think it's worth separating this into, say, three issues.

• The first issue is the issue of stability: that the python packages installed in /usr/lib/python3.X/site-packages, and the binaries installed in /usr/bin, change from release to release and we don't promise stability there. This forces you to update all the custom packages, which adds a ton of overhead effort and is really annoying.
• The second issue is how to install your own kinds of software on the device.
• The third issue is how to generate versions of the software you need that is properly compiled for the OT-2.

If that doesn't sound accurate or misses something, please let me know.

The third issue, generating versions of software you need - would it help if we or you generated the buildroot SDK? This creates a zip you can put somewhere and you activate its shell environment and that overrides all the environment markers that standard compilation and build tools use to find compilers and so on to use the buildroot toolchain. At that point, you could compile whatever you need, including python packages.

I don't know if a developer-mode build that is that different is a great idea because, quite frankly, I don't think we could support testing it and keeping it high quality.

Another path to solving the second issue would be extending the ability we already have to put stuff in /data. If you have a login shell then pip install should add stuff to a path in there. The robot server (and therefore protocols) won't load it by default to avoid something breaking the robot server, but you can add a sys.path manipulation to the protocol content.

As for the first issue, it's sort of bound up with the second. Unfortunately, the software preinstalled on the robot mostly exists for our software to run - we don't generally try and make the robot environment good for general purpose computing.

It kind of seems like a solution we could both live with is

• There's a full tree (/usr/bin, /usr/lib, /usr/lib/python3.X/site-packages) in the /data partition - you can do this right now and call programs installed there by full-path, or alter something to add it to PATH (and we can ship that for login shells)
• We make the buildroot sdk available so you can build software for the robot

Does that make sense?

[pbeaucage]

Thanks for the quick and comprehensive reply!

Yes, the buildroot sdk is exactly what I was thinking of, and sounds like exactly what's needed. I already maintain my own Python site-packages directory on these robots (pip --user will helpfully put the packages it fetches into the /data/ partition) so this is just a small extension of an existing workflow. To summarize my understanding: Basically, I then can distribute a zip file that contains precompiled binaries for a Python environment that lives in /data/ and a setup script that just downloads and unzips that file. I still need to recompile that zip for each new buildroot/OS update, but I can have tooling that uses the buildroot sdk to do that in a straightforward manner without relying on the specific package selections made in the OT2 firmware image.

Is it a straightforward ask for you to distribute the buildroot sdk? I imagine I could figure out how to generate it from the docs and docker scripts in this repo, but if you already use this and distributing it is easy enough, it would save me some effort. It might also help other users who want, for instance, scipy.

One, tangential question: does the OT-3 run a buildroot linux OS as well? I've seen bits and pieces of a lot more industrial-grade controls architecture with CAN bus and similar, curious how much things have changed. I don't mind telling users of my platform to just keep buying OT-2s but somebody is eventually going to ask. Just curious how future-proof this strategy is.

[sfoster1]

Yeah, I think it should be pretty easy for us to build the buildroot SDK. I think the first time we'd release it officially officially would be for 7.3 (no timeline on that yet) but we can make interim builds accessible probably.

I think your plan for the workflow definitely does work; the only thing to be cautious of is making sure whatever entrypoitns you use set PATH, LD_LIBRARY_PATH/other dynamic loader config, and PYTHONPATH appropriately.

The Flex uses openembedded, it's here: https://github.com/Opentrons/oe-core . That's because of different vendor support - rather than being based on a raspberry pi, it's based on a system-on-module from Toradex. It's also arm64 rather than the pi 3's armv7l, which is a much more widely-supported architecture - I've found that prebuilds are almost always available for that arch.