SurfEmb

Rasmus Laurvig Haugaard, Anders Glent Buch

SDU Robotics, University of Southern Denmark

CVPR 2022 | CODE | PDF

Abstract

We present an approach to learn dense, continuous 2D-3D correspondence distributions over the surface of objects from data with no prior knowledge of visual ambiguities like symmetry. We also present a new method for 6D pose estimation of rigid objects using the learnt distributions to sample, score and refine pose hypotheses. The correspondence distributions are learnt with a contrastive loss, represented in object-specific latent spaces by an encoder-decoder query model and a small fully connected key model. Our method is unsupervised with respect to visual ambiguities, yet we show that the query- and key models learn to represent accurate multi-modal surface distributions. Our pose estimation method improves the state-of-the-art significantly on the comprehensive BOP Challenge, trained purely on synthetic data, even compared with methods trained on real data.

Estimated Correspondence Distributions

Hover over the input image (left) or query image (middle) to see the estimated correspondence distribution. Drag the keys (right) to rotate them, or drag the slider to interpolate between keys' object coordinates and UMAP projection.

Interesting examples include discrete rotational symmetry: tless 2, tless 27, tless 23, itodd 7, full rotational symmetry: tless 14, tless 15, tless 17, and no symmetry: tless 7, tless 25, lmo 8, tudl 1, icbin 1, hb 1, ycbv 1.

The UMAP projection of keys shows the key manifold and primarily provides insight to constant ambiguities, like global symmetry. A query can also represent view-dependent ambiguities which is best seen by exploring the distributions.

Examples are chosen randomly for each object in the seven BOP Challenge datasets. Models are trained purely on synthetic images and shown on real images except for ITODD and HB, shown on synthetic images, because ground truth poses are not publicly available on the real images.

Distributions may appear worse than they are because of poor ground truth poses: lmo 1 (see the distributions near the edges), or poor meshes: tudl 1.

Pose Estimation

We use the distributions to sample, score and refine pose hypotheses. Our pose estimation method is trained and evaluated on the seven BOP Challenge datasets:

Synth: Method is trained purely on synthetic images. Note that only T-LESS, TUD-L and YCBV provide real images for training. * Methods do not use the available training data.