Combining CLIP semantic features with query-conditioned geometric priors to localize interaction regions in 3D point clouds — no training data required. Evaluated on the LASO benchmark (CVPR 2025).
CLIP-only methods lift 2D patch features into 3D and score points by text similarity. We multiply this semantic score by a geometry score derived from local surface descriptors — encoding the physical common sense that, e.g., graspable surfaces have consistent normals and slight curvature, while placeable surfaces are flat and upward-facing.
CLIP-only achieves higher P@50 (finds the right single points), but our geometry-aware fusion wins on IoU and SIM across all four affordance categories — producing better-shaped regions. For robotics this matters: a robot needs a graspable surface, not just the highest-scoring point.
Multiplying semantic and geometric scores enforces that both must agree — a point scores high only if it both looks right (CLIP) and is shaped right (geometry). Additive fusion lets high semantic scores compensate for geometrically implausible regions, which degrades IoU.
For the support affordance, "region for support" achieves P@50=0.960 vs. 0.340 for the full descriptive query — while IoU is stable across all phrasings. This suggests CLIP's text encoder is sensitive to query verbosity for peak localization, but geometry provides stable signal regardless. Practical implication: use short action-centric queries.
| Variant | P@50 | IoU | SIM |
|---|---|---|---|
| (A) CLIP only | 0.382 | 0.262 | 0.261 |
| (B) Geometry only | 0.332 | 0.295 | 0.292 |
| (C) Additive fusion | 0.327 | 0.294 | 0.291 |
| (D) Multiplicative (ours) | 0.323 | 0.354 | 0.325 |
| (E) Harmonic fusion | 0.329 | 0.353 | 0.328 |
| Query | P@50 | IoU |
|---|
Short action-centric queries dramatically outperform verbose descriptions on P@50, while IoU remains stable — suggesting geometry provides robustness to prompt phrasing.