Background
Related work
Method: Hybrid 4-dimensional augmented reality
Overview
Technical approach
-
A 3D reconstruction component runs on the server on a multi-core CPU and GPU. This component generates a 3D point cloud from the initial base images through feature extraction, matching, and the SfM procedure.
-
A user localization component runs on the server. This component takes a single photograph taken from a mobile device as input and derives the 3D position and orientation of the mobile device with respect to the 3D point cloud by solving a Direct Linear Transform equation followed by a Levenberg-Marquardt optimization against the underlying point cloud model.
-
A client component, which is a small program that runs on Android and iOS smartphones, sends user-captured images to the server. It also has the capability of drawing cyber objects on top of the photograph once it gets localization results from the server.
3D reconstruction with HD4AR
Feature Detection and Extraction
Feature Matching
Structure-from-Motion (SfM)
High-precision augmentation with HD4AR
Localization and augmentation
3D annotation
Results and discussion
Platform specification and data sets
Data set | Camera | Number of base images | Image size |
---|---|---|---|
Parking garage | Google Nexus S | 143 | 2560 × 1920 |
Center for the arts | Nikon D300S | 125 | 2144 × 1424 |
Norris hall | Google Nexus S | 50 | 2560 × 1920 |
Patton hall | Samsung Galaxy Nexus | 44 | 2592 × 1944 |
Performance of 3D reconstruction
Data set | System | Number of registered images/ | Elapsed time |
---|---|---|---|
Number of base images | |||
Parking garage | HD4AR with FREAK | 125 / 143 | 36.73 mins (×17.25) |
HD4AR with SURF | 138 / 143 | 42.01 mins (×15.08) | |
Bundler package* | 143 / 143 | 633.63 mins (×1) | |
Center for the arts | HD4AR with FREAK | 125 / 125 | 9.25 mins (×19.67) |
HD4AR with SURF | 125 / 125 | 12.49 mins (×14.57) | |
Bundler package* | 125 / 125 | 181.95 mins (×1) | |
Norris hall | HD4AR with FREAK | 50 / 50 | 2.62 mins (×27.53) |
HD4AR with SURF | 50 / 50 | 3.27 mins (×22.06) | |
Bundler package* | 50 / 50 | 72.13 mins (×1) | |
Patton hall | HD4AR with FREAK | 37 / 44 | 3.78 mins (×34.71) |
HD4AR with SURF | 43 / 44 | 4.80 mins (×27.34) | |
Bundler package* | 44 / 44 | 131.22 mins (×1) |
Performance of localization
Data set | System | Localization success-ratio | Average localization time |
---|---|---|---|
Parking garage | HD4AR with FREAK | 100% (50 / 50) | 5.42 sec (×19.94) |
HD4AR with SURF | 100% (50 / 50) | 6.42 sec (×16.84) | |
18.45 sec (×5.86)** | |||
Bundler package* | 100% (50 / 50) | 108.10 sec (×1) | |
Center for the arts | HD4AR with FREAK | 95.38% (62 / 65) | 3.17 sec (×29.40) |
HD4AR with SURF | 98.46% (64 / 65) | 3.47 sec (×26.86) | |
16.02 sec (×5.82)** | |||
Bundler package* | 100% (65 / 65) | 93.20 sec (×1) | |
Norris hall | HD4AR with FREAK | 100% (25 / 25) | 4.98 sec (×7.51) |
HD4AR with SURF | 100% (25 / 25) | 10.74 sec (×3.48) | |
13.31 sec (×2.81)** | |||
Bundler package* | 100% (25 / 25) | 37.38 sec (×1) | |
Patton hall | HD4AR with FREAK | 100% (25 / 25) | 6.33 sec (×5.10) |
HD4AR with SURF | 100% (25 / 25) | 10.07 sec (×3.20) | |
24.56 sec (×1.31)** | |||
Bundler package* | 100% (25 / 25) | 32.26 sec (×1) |
Discussion and research challenges
-
Quantifying the accuracy of image-based localization in terms of re-projection error to validate how cyber objects are precisely overlaid on top of real-world photograph.
-
Quantifying the quality of 3D point cloud, which will guide users to take a minimal number of images from various sites for initial bootstrapping, e.g. 3D reconstruction.
-
Further increasing the speed of localization by using supplemental information such as mobile GPS available in mobile devices to reduce data set to be matched. Minimizing the image resolution to reduce matching time is also in our focus.