Challenge:
The purpose of this challenge is to compare different methods for object pose estimation in a realistic setting featuring heavy occlusion. Our dataset includes eight objects in a cluttered scene. Given a RGB-D image the method has to estimate the position and orientation (a total of six degrees of freedom) of each object. You can participate by applying your method to our data and submitting your results. We will evaluate submitted results according to multiple metrics and display the scores for comparison.
Scores:
| Method | AD | 5cm, 5deg | IOU |
|---|---|---|---|
| Learning 6D Object Pose Estimation | 56.58% | 22.74% | 62.02% |
Dataset:
NOTE: Below you find the version of the occlusion dataset as it was used in our ICCV15 challenge. However, we released a reworked version of the dataset as part of the BOP Challenge. The reworked version contains all data (images, poses, 3D models of objects) and some annotation errors have been corrected. We advise to use the reworked version of the dataset.
You can find our dataset here. Please cite [Brachmann2014] and [Hinterstoisser2012] when using it. Brachmann et al. provided additional annotations for a RGB-D sequence originally published By Hinterstoisser et al. Description of file formats and folder structure can be found here. We provide the dataset under the CC BY-SA 4.0 license.
Evaluation:
We calculate the percentage of correctly estimated poses. We use three different criteria:
- The AD criterion [Hinterstoisser2012]: We calculate the Average Distance (AD) between all vertices in the 3D model of the part in the estimated pose and the ground truth pose. A pose is considered correct, when this average distance is below 10% of the object diameter.
- 5cm, 5deg [Shotton2013]: A pose is considered correct when the translational error is below 5cm and the rotational error is below 5deg.
- The IOU criterion: We calculate the 2D axis aligned bounding boxes of the part in the estimated pose and ground truth pose. We calculate the IOU (Intersection Over Union) of the bounding boxes. A pose is considered correct, when this value is above a threshold of 0.5.
How to participate?
In order to participate you have to:
- Download the dataset.
- Apply your method. You can use anything as training data except the test sequences provided in the dataset.
- Write your pose estimates to .info text files using the exact format as in the dataset. It is described here in Section 2.2.
- Compress your results to a single tar.gz file. Use the same folder structure as in this sample.tar.gz.
- Email your results to occlusion-challenge<at>cvlab-dresden<dot>de. Please provide the name of the method as well as a URL pointing to your project page or publication.
References
[Hinterstoisser2012]: Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary R. Bradski,
Kurt Konolige, Nassir Navab:
Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes. ACCV 2012
[Shotton2013]: Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, Andrew Fitzgibbon:
Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images. CVPR 2013
[Brachmann2014]: Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, Carsten Rother:
Learning 6D Object Pose Estimation using 3D Object Coordinates. ECCV 2014
[Michel2015]: Frank Michel, Alexander Krull, Eric Brachmann, Michael. Y. Yang, Stefan Gumhold, Carsten Rother:
Pose Estimation of Kinematic Chain Instances via Object Coordinate Regression. BMVC 2015.