EgoDelta: From Egocentric Scene Change to 3D Attachment Targets for Agricultural Manipulation

Quantitative results

Headline numbers from the paper's main tables: 30 tomato-vine scenes, 60 harvesting and deleafing interventions, 160 attachment targets and 180 physical robot trials. Hover a bar for the per-setting breakdown.

How EgoDelta works

A completed removal leaves a persistent change in the scene. EgoDelta compares 3D reconstructions from before and after the action to find what was removed and where it was attached, then turns that target into robot supervision. No attachment-target annotations are needed.

1. 01

   Egocentric pre/post demonstration

   A worker wearing Project Aria glasses records one continuous removal. Aria MPS provides camera poses and semi-dense scene points; frames sampled before and after the action form two view sets that need not share viewpoints.

2. 02

   2DGS reconstruction

   The plant is reconstructed independently before and after removal with 2D Gaussian Splatting in a common scene frame, so geometry removed by the action is absent from the post-action model.

3. 03

   Extract attachment target

   Each pre-action surface primitive is scored by its geometric and appearance mismatch, weighted by how reliably it was observed. Continuous scores and remaining-surface evidence then locate the former connection.

4. 04

   Robot trajectory generation

   The target anchors the robot's contact point while tracked hand motion supplies approach and manipulation directions, yielding feasible trajectories that also supervise a target-conditioned policy.

Removed-component localization

Qualitative comparison on one harvesting and one deleafing example, with the RGB overlay on top and the binary prediction below. Competing methods often respond to foliage or reconstruction artifacts, while EgoDelta keeps the thin extent of the removed component.

From demonstration to robot motion

Each tab follows one demonstration recorded with Project Aria glasses. The middle panel illustrates what EgoDelta does with it: reconstruct both scene states, score the removed component, place the attachment target, and plan a robot trajectory that reaches it.

Robot execution

A fixed-base RB-Y1 with one 7-DoF arm and an interaction-specific end effector, tested on held-out arrangements disjoint from the perception benchmark. A trial succeeds only if the intended component is removed without contacting another target, damaging the plant, triggering a safety stop, or requiring intervention.

Learned policy on the robot

The target-conditioned policy acting from the robot's own RGB-D observations.

Citation

@inproceedings{egodelta2027,
  title     = {{EgoDelta}: From Egocentric Scene Change to 3D Attachment Targets for Agricultural Manipulation},
  author    = {Koh, Nam Ho and Jang, Junyoung and Oh, Chaeyoung and Yoo, Changwoo and Lee, Jongho and Kim, Kee-Eung},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}