Single Dispense Undefined Offset Issue

[CamilloMoschner]

Hi everyone,

I found that when I dispense into multiple wells in the same column using multiple channels simultaneously the pipetting accuracy is exactly as desired:

for x in tqdm(range(88, 93,8)):
    dest_wells = prep_plate_0.children[x:x+5]

    await lh.dispense(
                      dest_wells, 
                      vols=[8]*len(dest_wells),
                      swap_speed = [20]*len(dest_wells),
                      homogenization_speed = [20]*len(dest_wells),
                      offsets = [Coordinate(x=0,y=0,z=1.66)]*len(dest_wells),
                      mix_speed = 20
    )

dest_wells
> [Well(name=prep_plate_0_well_11_0, location=(108.500, 070.000, 000.750), size_x=9.0, size_y=9.0, size_z=11.3, category=well),
 Well(name=prep_plate_0_well_11_1, location=(108.500, 061.000, 000.750), size_x=9.0, size_y=9.0, size_z=11.3, category=well),
 Well(name=prep_plate_0_well_11_2, location=(108.500, 052.000, 000.750), size_x=9.0, size_y=9.0, size_z=11.3, category=well),
 Well(name=prep_plate_0_well_11_3, location=(108.500, 043.000, 000.750), size_x=9.0, size_y=9.0, size_z=11.3, category=well),
 Well(name=prep_plate_0_well_11_4, location=(108.500, 034.000, 000.750), size_x=9.0, size_y=9.0, size_z=11.3, category=well)]

However, as soon as I use a single dispensation the arm moves about 5mm to the right and the tip then crashes into the plate.

await lh.dispense(dest_wells[channel_idx],
                          vols = [8],
                          swap_speed = 40,
                          homogenization_speed = 20,
                          flow_rates = 120,
                          blow_out_air_volume = 2,
                          offsets = [Coordinate(x=0,y=0,z=1.66)],
                          mix_speed = 30,
                          use_channels = [channel_idx]
                          )


channel_idx
> 0
dest_wells[channel_idx]
> Well(name=prep_plate_0_well_11_0, location=(108.500, 070.000, 000.750), size_x=9.0, size_y=9.0, size_z=11.3, category=well)

I have checked the specified destination wells and they are exactly the same (shown above with the associated code).

I observe the exact same behaviour at aspirations.

Here is one example that I filmed:
Working coordinates for multi-aspirate

source_wells = reagent_plate_1.children[:4]

for x in range(1):
    await lh.aspirate(
        source_wells,
        vols=[2]*len(source_wells),
        swap_speed = [50]*len(source_wells),
        homogenization_speed = [20]*len(source_wells),
        lld_mode = [lh.backend.LLDMode(1)]*len(source_wells),
        use_channels = [channel+4*x for channel in [0,1,2,3]]
    )

Disfunctional coordinates for single-aspirate

await lh.aspirate(
    source_wells[0],
    vols=[2],
    swap_speed = [50],
    homogenization_speed = [20],
    lld_mode = [lh.backend.LLDMode(1)],
    use_channels = [0]
    )

these two cells executed right after one another:

This is a very consistent across different locations and different plates.

Does anyone have an idea why single dispensations would suddenly change their x offset without any commands to do so (as far as I can see)?

[rickwierenga]

Sorry for the late reply.

This is because for single-resource dispenses, the 2d center of the resource is automatically added as an offset in LiquidHandler - useful when aspirating from a tube or beaker. This is not done when a list of resources is passed, even if the list contains one item.

In your case, the well is interpreted as a single-resource dispense which I had initially intended for tubes/beakers. STAR also adds this offset when aspirating from/dispensing to a Well → you see double the offset. If you had passed the well in a [] you would see correct behavior.

I did the single-resource aspirate/dispense after the list-dispense, and was not thinking clearly on how to disambiguate the API. Let’s do that now. I think we should either

1. always add the 2d center in LiquidHandler so we never have to do this in STAR. This offset would be added to any user-defined offset passed to the offsets parameter of LH. The sum would be sent to the backend. On the Opentrons, we would need to subtract the ‘default’ 2d center offset because their API already automatically centers an aspiration in a well.
2. always have STAR add the offset, remove from LiquidHandler. I think there is an argument to be made for having backends specify the center offset; this is effectively what already happens on Opentrons.

Both are equally ‘dangerous’ by symmetry. I think I like the first option better because it is super explicit. In the standard form, you will just see a resource and the offset to its left front bottom. The opentrons backend will have to adjust for that.

[CamilloMoschner]

Thank you for your explanation, @rickwierenga.

Just to make sure I understand:

1. the backend, in this case STAR.py, always calculates the 2D center and adds it to each aspiration/dispensation position, no matter whether the position is given inside a list or directly:

await lh.aspirate(pos1, vols=[5]) # position given directly
await lh.aspirate([pos1], vols=[5]) # position given inside a list

2. but, importantly, the liquid_handler calculates the 2D center and adds it to each aspiration/dispensation position, only if the position is given directly:

Which means…

await lh.aspirate(pos1, vols=[5]) # position given directly

…actually receives the 2D offset instructions twice, moving the aspiration/dispensation into the back, right corner of the container/well (which, of course, doesn’t even exist in the case of circular cross-section containers, and the tip is destined to crash into the plate… )

?

That makes everything I have observed make sense now.

As you said, the simple, direct solution is that all aspiration/dispensation positions must be given to the corresponding method (.aspirate(), .dispense()) inside a list and this should probably be placed into the docs with a big warning sign.

In regards to your two proposed solutions:
I also think solution 2 - have all 2D center offsets added only by the backend - is preferrable.
If the Opentrons API already performs this action it is likely that other machines that might get integrated in the future will do the same. Keeping aspiration positions consistent and predictable is very important. In this situation one should be able to look up the backend definitions and doesn’t have to worry about anything else to figure out what is happening.

[rickwierenga]

Exactly right.

@CamilloMoschner

As you said, the simple, direct solution is that all aspiration/dispensation positions must be given to the corresponding method (.aspirate(), .dispense()) inside a list and this should probably be placed into the docs with a big warning sign.

People don’t read the docs. I will make the API sensible.

@CamilloMoschner

In regards to your two proposed solutions:
I also think solution 2 - have all 2D center offsets added only by the backend - is preferrable.
If the Opentrons API already performs this action it is likely that other machines that might get integrated in the future will do the same. Keeping aspiration positions consistent and predictable is very important. In this situation one should be able to look up the backend definitions and doesn’t have to worry about anything else to figure out what is happening.

Probably option 1 is more predictable IMO. It is more intuitive to always just aspirate from the center (on the xy plane). Imagine calling aspirate on a beaker/bucket and it aspirates from the corner…

Passing a single well should be treated the same as if that well were passed in a list. This cannot be cleanly reconciled with aspirating from the corner (not-center) if not a well.

[rickwierenga]

One other thing I considered: if we go with option 1 of passing the offset to the backend go to the center of a resource, for every resource in the command, we run into this issue: offsets on LiquidHandler will mean offsets from center of the resource (which is the default location) and in the call to the backend offsets will be from the lfb of a resource. This is extremely dumb. One way around this is to have offsets in LiquidHandler also be from lfb and just default to the center, meaning that if the user specifies an offset, this is to lfb - also extremely dumb.

For this reason, let’s go with option 2. offsets, both in the call to LH and the call to the backend. Backends will be responsible for adding the “extra center offset” (eg making xs in STAR be the resource.get_absolute_location + resource.center (for which we now have short hand), and additionally add the offsets passed in eg Aspiration.offset.

Will fix soon.

[rickwierenga]

deprecate single values in LH

[CamilloMoschner]

@rickwierenga, just so I understand:

Before this PR, only aspiration/dispensation commands targeting a Well took offsets in regards to the center-center-bottom of the well,
but when aspirating/dispensing from a different Container or its other subclasses like Tube and Trough offsets was in regards to the front-left-bottom of the Container?

(I can say that this has until today been the case for Tube and Trough)

[rickwierenga]

it should have been wrt the center of the tube of a trough, and definitely should be after this PR. Please let me know if that’s not the case.

The difference is that before, STAR would only add the center of a resource automatically if the resource was a Well or a TipSpot. This meant that to aspirate from the center of anything else, you’d have to specify that in the Aspiration.offset attribute (smh). When aspirating from a single trough, LH would do this → causing the problem when you pass a single Well: LH treated that as a single resource and would by default add the center as an offset. In addition, LH would add on top of that the user-specified offset.

Now, LH only adds offsets wrt the center of the resource, which is most cases is None (after this PR I will make it non-optional and use Coordinate.zero()). STAR is always responsible for going to the center, regardless of whether something is a Well, TipSpot, Tube, Trough, etc.

[CamilloMoschner]

I see, this makes a lot of sense.

For trough and tube, up until this moment it did definitely require to state the offset in regards to flb !

This is the offset needed for the new Hamilton trough:

coordinates = [Coordinate(18.5, y, 0) for y in [98, 89, 80, 71, 45, 36, 27, 18]]

Now, this PR changed this, and if I update to the latest PLR version without carefully looking and understanding the changes, the machine will crash, my workflow will crash and my samples might be irreversibly lost.
I think we need to phase in this change slowly in regards to our latest discussion on giving users time to update their automation protocols to PLR changes that have the potential to break automation scripts - particularly this one: while other changes we made and discussed with @jkhales would immediately appear during the import of a resource (and therefore “just” represent a pain to figure out how to update computationally), this change will only be revealed mid-run when attempting to aspirate/dispense a real sample, i.e. will have a physical consequence.

[CamilloMoschner]

There is another aspect to consider:

When switching from …

@CamilloMoschner

coordinates = [Coordinate(18.5, y, 0) for y in [98, 89, 80, 71, 45, 36, 27, 18]]

…to the “new” / originally desired offsets (i.e. in relation to the 2D center), we now have to convert these values correctly:

• the x dimension (in this case) is the center, so it will have to change to 0,
• but the y dimensions will all have to be subtracted by the resource.center().y:

coordinates = [Coordinate(0, y-resource.center().y, 0) for y in [98, 89, 80, 71, 45, 36, 27, 18]]

…i.e. the coordinates for channels=[4,5,6,7] (in an 8-channel system) will now all take negative values?

[rickwierenga]

@CamilloMoschner

coordinates = [Coordinate(18.5, y, 0) for y in [98, 89, 80, 71, 45, 36, 27, 18]]

this is wrt the lfb? I wonder why that didn’t happen automatically

from the removed code:

[rickwierenga]

@CamilloMoschner

• but the y dimensions will all have to be subtracted by the resource.center().y:

coordinates = [Coordinate(0, y-resource.center().y, 0) for y in [98, 89, 80, 71, 45, 36, 27, 18]]

…i.e. the coordinates for channels=[4,5,6,7] (in an 8-channel system) will now all take negative values?

yes

but we already space the channels equally along the y-axis. from the new code:

[CamilloMoschner]

@rickwierenga

but we already space the channels equally along the y-axis. from the new code:

That would crash the tips in the case of the Hamilton trough:

1. the trough floor geometry is non-symmetrical, i.e. at low volumes tips have to be at the right locations or the ones auto-spaced above the chamfered wall will crash into the chamfered wall,
2. due to the anti-splash wall located in the y-center of the trough a tip auto-spaced into the center (i.e. for 1, 3, 5, 7 channels being used) will crash into the anti-splash wall (hence the jump in the list comprehension for the y positions in the middle of the the list)
3. I previously (tbf quite a while ago because I realised this is a pain) had a problem with the auto-spacing when targeting to channels to aspirate from the same 24-deep well plate well: the auto-spacing chose maximal spacing distance but did not factor in that it could not move closer than 4.5 mm to each wall of the well. So it moved to close to the walls on each side and scratched it.
   I had to manually switch it to:

""" Aspirate from 4x10ml containers with all 8 channels
"""
no_channels_used = 8
await safe_aspirate(
    lh,
    flatten([[well]*2 for well in source_well]),
    vols=[11]* no_channels_used,
    hamilton_liquid_classes = [Tip50ulFilter_DNA_DispenseSurface_Empty]*no_channels_used,
    offsets = flatten([[Coordinate(0, 4.5, 0), Coordinate(0, -4.5, 0)] for x in range(int(no_channels_used/2))])
)

…which is okay for testing but too hacky for production.

[rickwierenga]

tangent: it sounds like these offsets are “required” to physically use the Hamilton trough. Do you think this could be made a static attribute of this particular trough, so you could say HamiltonXXTrough.lh_offsets?

[CamilloMoschner]

Tangent continued: Yes, I thought about this but didn’t know how to implement it. Are you suggesting adding the specific offset to a Container to the definition of said Container as a special attribute that is defaulted to None?
Does Container.lh_offsets already exist?

• I really like this idea!

[rickwierenga]

I just wrote a quick test to double check that offsets are handled the same before and after the change:

  async def test_aspirate_single_resource_offset(self):
    self.lh.update_head_state({i: self.tip_rack.get_tip(i) for i in range(5)})
    with no_volume_tracking():
      await self.lh.aspirate(self.bb, vols=10, use_channels=[0, 1, 2, 3, 4], liquid_height=1,
                             offsets=[Coordinate(1, 1, 0)]*5)
    self._assert_command_sent_once(
      "C0ASid0002at0 0 0 0 0 0&tm1 1 1 1 1 0&xp04875 04875 04875 04875 04875 00000&yp2108 1971 "
      "1835 1698 1561 0000&th2450te2450lp2000 2000 2000 2000 2000 2000&ch000 000 000 000 000 000&"
      "zl1210 1210 1210 1210 1210 1210&po0100 0100 0100 0100 0100 0100&zu0032 0032 0032 0032 0032 "
      "0032&zr06180 06180 06180 06180 06180 06180&zx1160 1160 1160 1160 1160 1160&ip0000 0000 0000 "
      "0000 0000 0000&it0 0 0 0 0 0&fp0000 0000 0000 0000 0000 0000&av00119 00119 00119 00119 "
      "00119 00119&as1000 1000 1000 1000 1000 1000&ta000 000 000 000 000 000&ba0000 0000 0000 0000 "
      "0000 0000&oa000 000 000 000 000 000&lm0 0 0 0 0 0&ll1 1 1 1 1 1&lv1 1 1 1 1 1&zo000 000 000 "
      "000 000 000&ld00 00 00 00 00 00&de0020 0020 0020 0020 0020 0020&wt10 10 10 10 10 10&mv00000 "
      "00000 00000 00000 00000 00000&mc00 00 00 00 00 00&mp000 000 000 000 000 000&ms1000 1000 "
      "1000 1000 1000 1000&mh0000 0000 0000 0000 0000 0000&gi000 000 000 000 000 000&gj0gk0lk0 0 0 "
      "0 0 0&ik0000 0000 0000 0000 0000 0000&sd0500 0500 0500 0500 0500 0500&se0500 0500 0500 0500 "
      "0500 0500&sz0300 0300 0300 0300 0300 0300&io0000 0000 0000 0000 0000 0000&il00000 00000 "
      "00000 00000 00000 00000&in0000 0000 0000 0000 0000 0000&",
      fmt=ASPIRATION_COMMAND_FORMAT)

(based on TestSTARLiquidHandlerCommands.test_aspirate_single_resource, with offset added).

updated lh.aspirate syntax:

 await self.lh.aspirate([self.bb], vols=[10]*5, use_channels=[0, 1, 2, 3, 4], liquid_height=[1]*5, offsets=[Coordinate(1, 1, 0)]*5)

[rickwierenga]

tangent continued^2: initially I was suggesting this about the hamilton trough in particular, which would require a new class (allowed since it has added funcionality). But if we see this pattern repeat, perhaps we can look into adding it in a more generalizable way.

[CamilloMoschner]

@rickwierenga

I just wrote a quick test to double check that offsets are handled the same before and after the change:

I can confirm after testing directly on the machine:

1. This breaks the behaviour of aspirating/dispensing from a Trough that has been in place until yesterday: the tips are now moving off by exactly well.get_size_y()/2 → all automation protocols written until yesterday using offsets in a Container that is not Well are now broken.
   (I obviously tested this with the trough moved to the side by 2 rails, so as to still allow me to see where the tips go but not endanger my machine to violently crash, which it would have otherwise)
2. Targeting multiple pipettes to the same trough without declaring offsets still does not work with the latest update. Instead this error is returned:

await safe_aspirate(
            lh,
            source_wells_l=[current_buffer_well]*no_channels,
            vols=[transfer_target]* no_channels,
            use_channels = use_channels[:no_channels],
            hamilton_liquid_classes = [current_liquid_class]*no_channels,
            surface_following_distance=[20]*no_channels,
            # offsets = coordinates[:no_channels],
        )

→

File 
pylabrobot\liquid_handling\liquid_handler.py:1831, in LiquidHandler._trigger_callback(self, method_name, error, *args, **kwargs)
   1829   callback(self, *args, error=error, **kwargs)
   1830 elif error is not None:
-> 1831   raise error

File 
pylabrobot\liquid_handling\liquid_handler.py:764, in LiquidHandler.aspirate(self, resources, vols, use_channels, flow_rates, offsets, liquid_height, blow_out_air_volume, **backend_kwargs)
    762 error: Optional[Exception] = None
    763 try:
--> 764   await self.backend.aspirate(ops=aspirations, use_channels=use_channels, **backend_kwargs)
    765 except Exception as e:  # pylint: disable=broad-exception-caught
    766   error = e

File 
pylabrobot\liquid_handling\backends\hamilton\STAR.py:1515, in STAR.aspirate(self, ops, use_channels, jet, blow_out, lld_search_height, clot_detection_height, pull_out_distance_transport_air, second_section_height, second_section_ratio, minimum_height, immersion_depth, immersion_depth_direction, surface_following_distance, transport_air_volume, pre_wetting_volume, lld_mode, gamma_lld_sensitivity, dp_lld_sensitivity, aspirate_position_above_z_touch_off, detection_height_difference_for_dual_lld, swap_speed, settling_time, homogenization_volume, homogenization_cycles, homogenization_position_from_liquid_surface, homogenization_speed, homogenization_surface_following_distance, limit_curve_index, use_2nd_section_aspiration, retract_height_over_2nd_section_to_empty_tip, dispensation_speed_during_emptying_tip, dosing_drive_speed_during_2nd_section_search, z_drive_speed_during_2nd_section_search, cup_upper_edge, ratio_liquid_rise_to_tip_deep_in, immersion_depth_2nd_section, minimum_traverse_height_at_beginning_of_a_command, min_z_endpos, hamilton_liquid_classes)
   1400 async def aspirate(
   1401   self,
   1402   ops: List[Aspiration],
   (...)
   1443   hamilton_liquid_classes: Optional[List[Optional[HamiltonLiquidClass]]] = None
   1444 ):
   1445   """ Aspirate liquid from the specified channels.
   1446 
   1447   For all parameters where `None` is the default value, STAR will use the default value, based on
   (...)
   1511       pylabrobot/liquid_handling/liquid_classes/hamilton/star.py
   1512   """
   1514   x_positions, y_positions, channels_involved = \
-> 1515     self._ops_to_fw_positions(ops, use_channels)
   1517   n = len(ops)
   1519   if jet is None:

File 
pylabrobot\liquid_handling\backends\hamilton\base.py:351, in HamiltonLiquidHandler._ops_to_fw_positions(self, ops, use_channels)
    349       continue
    350     if abs(y1 - y2) < 90:
--> 351       raise ValueError(f"Minimum distance between two y positions is <9mm: {y1}, {y2}"
    352                        f" (channel {channel_idx1} and {channel_idx2})")
    354 if len(ops) > self.num_channels:
    355   raise ValueError(f"Too many channels specified: {len(ops)} > {self.num_channels}")

ValueError: Minimum distance between two y positions is <9mm: 1240, 1240 (channel 0 and 1)

[CamilloMoschner]

I can also confirm that this change breaks Tube, in addition to Trough

[rickwierenga]

Thank you for checking this in base reality.

I think I see what the issue is: you specify the single resource that you want to aspirate from no_channels times ([current_buffer_well]*no_channels) whereas I just did it once (eg [self.bb]). In both our cases we used use_channels to specify which channels should be used. This caused the if len(resources) == 1: check in the updated code to not get hit.

We should choose which of our approaches we should use, we can have 1 max. Unless you disagree, I think your way of specifying it (with the same resource repeated n times) is more reasonable.