Neural Textures: From Robust 3D Reconstruction to Animation Synthesis

Summary: This paper proposes neural textures, a high-dimensional feature map that encodes
more information than just colors. The paper also proposes the Deferred Neural Renderer to
train neural textures, enabling end-to-end training using the rendering network. The texture is
extracted from the first 3 layers of the neural texture, for which an immediate re-rendering loss is
applied to reinforce color representation, and applied to a precomputed uv map for some 3D
model. This scene is rendered and photometric loss is calculated. The paper also demonstrates
how this can be used for novel view synthesis as well as controlled animation synthesis for
videos.

Strengths: This method seems to outperform previous methods in terms of visual fidelity and
fine details. This is enabled through the neural texture hierarchy which is an analog to texture
mipmapping. By applying increasing levels of regularization, some layers learn coarse details
while other layers learn fine details. This method can also be used in conjunction with
Face2Face to do animation synthesis, essentially creating a deepfake of the target actor in a
video. The author rightfully highlights the ethical concerns with this capability.

Weaknesses: The proposed technique requires a given geometry and a corresponding uv map.
This is very limiting since it depends on other techniques to reconstruct the 3D geometry and
precompute a corresponding uv map. The paper doesn’t explore more complex models that
would leverage multiple texture maps. In that case, it would make sense for the neural texture to
have an increase in dimensions, but doesn’t explore how that would scale.

Reflection: The most insightful thought about this paper is that by learning a high-dimensional
neural texture, including the regularization and immediate re-rendering loss, a given model and
precomputed uv map can use the learned texture with state-of-the-art visual fidelity. The model
is trained with a neural deferred renderer, proposed by the paper, which renders the scene and
calculates the photometric loss. The network is relatively simple and the idea of neural textures
is also simple, making it easy to combine with other techniques or models. The next step for this
research could be to attempt this method with multiple objects that require multiple textures and
research how to scale this model.