Boundary-aware 3D Building Reconstruction

from a Single Overhead Image

Jisan Mahmud, True Price, Akash Bapat, Jan-Michael Frahm

{jisan,jtprice,akash,jmf}@cs.unc.edu

UNC-CS 1
Boundary-aware 3D Building Reconstruction from a Single
Overhead Image

q Goal: Develop a learning framework to model buildings in 3D from

a single overhead image in a boundary aware fashion.

q Challenges:
o Buildings are often densely packed -
this leads to overlapped detections.
o Building height prediction is ill-posed
from a single image.
o Parametric modeling limits building
modeling ability of existing methods.

UNC-CS 2
Contributions
Given an overhead image we propose:

q Multi-task network to jointly learn building detection, modified boundary SDF

function (BPSH), pixel-wise height (nDSM), and semantic labeling

q Multi-feature formulation: features of BPSH, nDSM, semantic label prediction

augments the building detection features

q Detection overlap refinement technique using BPSH

q Instance-level reasoning to improve BPSH for better overlap refinement

UNC-CS 3
Multi-task, multi-feature network

Building Outline Proposal

Module (Mask R-CNN) Overlap Refinement

Modified Signed Distance

Function (BPSH) 3D Building Modeling

Pixel-wise Height
Prediction

Single Overhead Feature

Image Extractor
Pixel-wise Semantic
Labeling

UNC-CS 4
3D building modeling steps
1. Jointly learn building outline detection, BPSH, nDSM, and semantic labeling
using-
𝐿𝑜𝑠𝑠 = 𝛼! 𝐿"#$%&'( + 𝛼) 𝐿*+,- + 𝛼. 𝐿'/,0 + 𝛼1 𝐿,(0
2. (optional) use building outline detection to improve BPSH using instance level
reasoning.

3. Refine overlapping proposals in building outline detection using BPSH.

4. Use median nDSM of each predicted building outline as the height of that
building.

UNC-CS 5
 Building Outline Proposal
Module (Mask R-CNN)

Modified signed distance function - BPSH Modified Signed Distance

Function (BPSH)

Pixel-wise Height

Ø Boundary Proximity Signed Heatmap (BPSH) Single Overhead

Image
Feature
Extractor
Prediction

Ø Defined from building boundaries

Pixel-wise Semantic
Labeling

Ø Ternary behavior sharply distinguish the building boundaries

BPSH

Distance from nearest building boundary Buildings in BPSH (green: boundary,

overhead image red: inside, blue: outside)

UNC-CS 6
 Overlap refinement

Initial building Predicted BPSH Score and NMS using detection Recover the proposals Final building
proposals with overlaps confidence and BPSH lost during NMS detections

UNC-CS 7
Qualitative results

Overhead Wang and Srivastava Our building Ground-truth Our 3D

Image Frahm et al. outlines building outlines Buildings

UNC-CS 8
Qualitative results – 3D building models

UNC-CS 9
 Quantitative results
Wang and Srivastava Mou and
Datasets Methods Ours
Frahm et al. Zhu
Building Outline F1 68.34 57.86 - -

GRSS DFC Median Height RMSE 2.79 2.94 3.59 3.19

2019 Sem. Seg. F1 (Building) 94.2 - 76.8 -
Sem. Seg. F1 (Ground) 95.2 - 92.6 -
Building Outline F1 71.98 57.97 - -
Median Height RMSE 2.75 2.88 3.59 3.48
Potsdam
Sem. Seg. F1 (Building) 97.31 - 93.93 -
Sem. Seg. F1 (Impervious) 92.09 - 87.27 -

UNC-CS 10
Thank you!
q Check out our paper-
Boundary-aware 3D Building Reconstruction from a Single Overhead Image

UNC-CS Supported by NSF grant No. IIS-1816148 11