User feedback on Schedulers

Schedulers
1

Hi everyone,

I am looking into possibilities for orchestration and scheduling (both open-source / script-based and commercial). For commercial solutions, Green Button Go, Cellario and Overlord seem to come up quite often. These software (or scripts) would be mostly used for larger automated workflows with multiple devices (Liquid handlers from all size and brands, cobots, etc…). I am especially interested in the deployment speed, breadth of device integration, and end-user experience.

Is there other solutions that you know work well for your automation workflows, or do you have some feedback (+/-) relative to the use of GBG, Cellario or Overlord ?

Cheers !

2

In my opinion the moat these companies have is mostly around licensing device drivers. I would like to see open-source scheduler solutions that allow for more flexibility and customizability. There’s such a wide range of scientific workflows (which are themselves continuously changing and advancing) that closed-source solutions are not really able to keep up. There is a reason that basically every popular software development tool is open source, and that all big tech companies invest heavily into open source.

Other companies that invest into open source are Toyota, McDonalds, and Walmart, so it is not just a tech thing. Of course, there has to be strategy associated with the approach, it is not just giving away free stuff. There are some companies that would benefit massively from an open source model in automation and some companies that would lose out if that model becomes the standard.

I don’t really think every automation engineer should have to become a software developer for this to work. Automation code should be easier for software devs to work with and understand, so they can create tools that help automation engineers. This is easiest when automation scripts are in the same language/ universe as dev tools familiar to software people, like Python.

5 Likes
3

Before we go off on this matter, some of the folks over at BiB put together this spreadsheet with TechBio Tools in mind. Note that the section for Automation is missing a lot of players… with that said, it’s still handy.

3 Likes
4

Hi @Jeremy,
When it comes to “Orchestration” Biosero is defiantly a market leader in that regard and has a wide array of software tools that provide connectivity of devices at an integrated workcell level and orchestrated solutions at a laboratory level. I am not on this forum to make sales pitches but to answer the questions above:

Green Button Go Scheduler and Orchestrator:
-We routinely deploy large complex integrations featuring multiple integrated workstations, stores, etc.

-Biosero is 100% hardware agnostic, we work with the best devices from robotic arms, liquid handlers, and peripheral equipment.

-Biosero has 400+ existing device drivers and a team dedicated to creating new as customer needs requires. We can develop a green button go driver as long as there is an API, and the device is robot friendly (loadable with a robotic arm).

-Deployment time depends on the scale of the project and ability to get components needed. A standard workcell can be delivered in 4-6 months.

-As far as references happy to connect with current users, but at SLAS, several of our customers are speaking at “Adapting to Demand by Orchestrating Your Lab”. This talk is on Monday Febuary 27 at 12:00PM. It will feature speakers from Biosero, Invitate, DeepCure, and Metagenomi. If you can attend this would be a great showcase to show you a wide array of integrated solutions Biosero has deployed.

-Easy to use low code/no code interface, but have capability of easily integrating scripts from C#, Python, VBA, etc. It can be easy as a drag and drop interface or some people prefer to make things more complex implementing custom solutions using scripts.

@Jeremy if you are interested in a Demo or learning more, please send me a DM and I can connect you with the correct regional team. I am also happy to connect in person at SLAS if you will be attendance next week.

Take Care,
Derek

2 Likes
5

Stefan,

I’m sure you saw this blog but it’s a great starting point for discussion around the current landscape of open-source development in the life sciences.

Open Development of Scientific Software

There are some good examples of projects, pitfalls of the realities of open source but also a shout out to PyLabRobot.

3 Likes
6

Yeah! I did see it, good stuff. I’m actually writing a perspective paper about it as well so my gears have been turning on the topic.

7

Oh interesting, I’m definitely curious about your thoughts on the matter.

Maybe there’s a good opportunity for an open discussion (a virtual round table of sorts), I’m sure people in the community will listen in and participate in such a conversation.

2 Likes
8

I agree Stefan, open source is the way to go, both for for-profit and non-profit projects. I struggle at the moment to find alternative able to keep up with big names in the scheduling space though.
A PyLabRobot sister project on Scheduling and device integration would be nice indeed !!!
Out of curiosity, if or when you integrate multiple devices, what do you go for ? some particular Python libraries or routines that work best in your applications?

2 Likes
9

Thank you for the resource, Luis, that’s very helpful (not only for the scheduling actually!)

10

Here is a small guide on how to integrate 3rd party (ie non-Hamilton) equipment into PyHamilton. The best case is a .NET dll that you can import with pythonnet.

https://forums.pylabrobot.org/t/getting-started-with-pyhamilton/28/5?u=stefan

As far as a scheduler project, it sounds interesting but I’d want to be more clear on what the needs are. To my knowledge, schedulers provide control logic around the operation of different pieces of equipment and software resources. When I write a Python script that talks to a robot and other integrated equipment, that seems to fulfill the basic use-case of a scheduler as I understand it.

If there’s additional functionality in commercial schedulers now that is not available as a Python package but that you’d like to be, I’d love to know about it. We’re always trying to come up with useful tools like that.

11

So I can only speak to end user experience here for the GBG product.
Like any it has had its ups and downs. The set up is pretty easy - we did this in Europe in 2018, before they had very much happening here. So there was some misses as talking to west coast USA from Denmark was kind of messy. But, that has been fixed and they have a full service team here in Europe. So from the point they come to install - depending on the complexity of the system this will take a few days. It is worth having one of your staff around to follow, help out, learn while they do this. Plan for at least a week.
Ours is hooked to 2 Echo’s , incubator, FACS machine, plate washer, dispenser, shaker, and plate reader (colour and flouresence), and centrifuge. That would be 7 different source manufactorers. This has not been really much of an issue. The drivers have worked for this well. The one exeption is the Echo’s - this is not fully smooth as they do not like being a slave, so in the most recent version of GBG we have had issues around the system being ‘stuck’ when the echo is running, with the arm now being stalled at the echo until the run is complete. But I have heard they are working to fix it, and it was not the case in the older version.
From a user standpoint, this is an easy software to get your head around. There is no need for coding - but there is space for that if you want to. So you can run this with pretty much a graphical interface and never do a line of code and run through multi-machine protocols easily.
There is also a set up where the user has access to only pre-made protocols and cannot mess with the behind the curtain stuff. This can be handy if you have techs that are a little put off by all the background stuff - or students you just want to prevent then from fucking stuff up.

So for our system there was hickups. But I would expect that you can have a moderate system set up and the users fairly comfortable with using the system and creating protocols in something like 4-6 months (obviously depending on your users comfort with automation, more if this is ‘scary’ for them).

5 Likes
12

I’m a Cellario user. I’ve also used VWorks to run a BioCel system. I don’t have experience with GBG or Overlord. Brooks (now Azenta) used to offer a scheduler called Sprint6; not sure if they still offer it. It was nice when I demoed it. Wako automation might offer scheduler as well.

We use our Cellario systems for high throughput screening. At last count our newest system has about 50 devices including Echos, a PheraStar and a Yokogawa CV8000 high content imager. I like a lot of things about Cellario. For many protocols you can drag-and-drop to create the workflow. We hired a new automation engineer and he was comfortable running Cellario in a few months and was creating protocols in a year or so. More complicated protocols often require scripting (which is C#) and that has a steeper learning curve.

I’m not sure if you are trying to buy the software alone, but I don’t think HighRes licenses Cellario apart from a system purchase.

My installs have all been mostly on schedule, setting up Cellario on a system takes time - if I recall correctly it was on the order of about 6 weeks from build to pre-FAT. I usually budget about 10-12 months from sending a PO to commissioning. Our last system upgrade suffered from 3rd party device build times due to component shortages.

There have not been many devices I was told I could not integrate. If it has an API they can probably build a driver. Not all the drivers work the way you think they might, and documentation on drivers is lacking. You can always open a ticket with the helpdesk to ask for help, though.

4 Likes
13

+1 my vote of support for open source scheduling software. There is a market for maximum flexibility, like a library of code, especially in Boston, SF, and far off reaches in the world where the big companies don’t offer support. It would require a diverse team of talented outlaws, geniuses, and artists to build such a product, though. The inventor of GBG left Biosero… maybe he’d be a resource to consider.

2 Likes
14

I think an open-source scheduler is a great idea, but I’m not sure about the need for it to be a dedicated project, at least in the world where people are using things like PLR for running instruments. It seems to me that the “scheduler” just emerges organically from continued development of PLR-based control for instruments, because all instrument control is centralized under the “universal language” of PLR. In that case, the only effort needed is towards getting more things runnable via PLR.

Of course maybe there are more sophisticated use cases for schedulers, but to my understanding and experience I agree with @Stefan:

To me the existence of most of these companies is simply because of the lack of standardized control for instrumentation, requiring third parties to create interface solutions between different manufacturers (aka schedulers.)

I also can’t help but say this caught my eye as completely insane:

Why should this be the case? It should be like…a few weeks to be creating protocols, I don’t care how sophisticated. It taking a year to me just speaks to how bloated and inefficient existing solutions are.

1 Like
15

I think this is mostly right, scheduling logic is pretty straightforward in most cases that people are now using a scheduler application for. I think there’s a lot of situations where it’s not though, and for labs that have multiple interconnected workcells or experimental parallelization within a single workcell, planning a feasible schedule can entail a very high number of combinations of possible paths. I bet there is still a lot of opportunity for open source libraries in this area.

I really feel this. Creating a biology workflow or pipeline is always going to be slower than pure software but a lack of open source dev tools is a major hindrance that we can move past.

2 Likes
16

It doesn’t take nearly this long. I was up and running comfortably creating any Cellario protocol I needed in just a couple of days. It’s a very intuitive scheduler. I would not say that Cellario is bloated in any way. But there are a lot of considerations for schedulers that are not same as for normal liquid handlers. When plates get introduced, how many of those plates can be active at any given time, limiting these flows is vital to designing protocols that work across any number of plates.

1 Like
17

I see some of these things being conceptually similar to the tracker for tips/liquids in PLR, just extended upward to tracking entire labware across instruments, rather than tips in a rack or liquids in a rack or other container, and as such completely extensible via PLR-like logic.

Sure it gets quite complex, but I can’t imagine anything insurmountable.

2 Likes
18

Ive been at HighRes for about 6 years now deploying Cellario systems and training and documentation is a big focus to help enable users to be successful. There is a wide variety of use cases and processes that Cellario systems and protocols can be designed to handle, so much so that it is difficult to directly compare onboarding from one system to another. I would echo @jnecr in saying that getting comfortable writing protocols in Cellario and general operation can take just a few days of being hands on with the software. The complexity scales with the workflows being run, devices being used, amount of labware being processed, timing restrictions, custom scripting and data tracking, the list goes on.

For an example of scale a simple system may have a protocol with the steps to move a plate from storage, dispense liquid, spin the plate in a centrifuge, take a measurement in a reader and return to an incubator. Where a more complicated workflow may have hundreds of steps detailing a process over days of runtime with strict timing requirements for materials being used. A bit of an iceberg in terms of complexity but I’ve found Cellario is very flexible in how it can handle each.

1 Like
19

Open source scheduler is not enough for lab automation, for we have to need more documents/informations/trainings/supports to use the scheduler and develop the driver for device. So I think the best value of existing scheduler vendors is services instead of software.

2 Likes
20

I feel like I have had the opposite experience. Creating a biology workflow or pipeline is easier than figuring out how to squeeze every ounce out of your software and/or hardware. If you’re doing dynamic workflows, it’s a pain. And you may be stuck with your software MVP in highly regulated environments. Furthermore if you’re a lab automation champion (meaning you push and promote new hardware) you’re going to be at the mercy of the problems that usually plague early adoption.

Anyone can write an aspirate/dispense/read command but writing and designing a robust, scalable and composable software stack takes way more time than putting together an NGS pipeline imo.

2 Likes