Four Dimensional Cultural Heritage World

Reporting

Project Information

4D-CH-WORLD Funded under

FP7-PEOPLE
Grant agreement ID: 324523
Overall budget
€ 1 360 092,79
Closed project

EU contribution
Start date End date
€ 1 360 092,79
1 January 2013 31 December 2016

Coordinated by
TECHNICAL UNIVERSITY OF
CRETE
Greece

Final Report Summary - 4D-CH-WORLD (Four Dimensional

Cultural Heritage World)
4D-CH-Word is a Marie Curie project funded by European Union with the main purpose of designing,
analyzing, researching and developing computer vision and machine learning algorithms for a cost-
effective creation of 4D digital maps in cultural heritage. 4D digital modelling implies the creation of precise
time-varying 3D reconstructions of cultural heritage objects to capture temporal geometric
variations/distortions, i.e. a spatio-temporal assessment.
The key research challenge for 4D modelling, was the data collection over heterogeneous unstructured
web resources. Such “in the wild” data include outliers and significant noise, since they have not been
created for 3D modelling and reconstruction purposes. In addition, GPS and geo-information is limited or
non-existent. However, such data allow for a massive reconstruction of the content even for monuments
that have been destroyed due to natural phenomena or humans’ interventions.

1 of 3
The project exploits digital information from both unstructured data, available over distributed multimedia
repositories and social networks, and from in-situ data acquisition procedures. The 4D digital modelling
methods were testing in Calw, a town in the middle of Baden-Württemberg, located 40 km southwest of
Stuttgart. Other demonstration scenarios refer to the digital reconstruction of the monument of Porta Nigra
in Germany and Padrão dos Descobrimentos in Portugal.
The first step involved the content-based retrieval of visual information from distributed multimedia
databases, like Picasa, Flickr and Twitter. Machine learning methods, exploiting unsupervised clustering
algorithms, were used to isolate data outliers and group together similar data with respect to their visual
properties. Additionally, textual analysis and processing take place to assign semantic metadata (tags) to
the visual collected data.
As far as the in-situ data collection process in concerned, terrestrial laser scanning, close-range
photographs and aerial imagery have been used. All these information are appropriately fused, using
computer vision algorithms, machine learning methods and photogrammetry tools for a cost-effective,
automated and precise time varying 3D modelling.
4D-CH-World supported various capturing procedures, including generation of dense 3D point clouds and
terrestrial and aerial overlapping images. In the context of data analysis, novel computer vision tools have
been adopted for fusing the heterogeneous collected data to improve the precision and accuracy of
modelling towards an automated framework. Finally, synthetic data and computer graphics methods are
exploited to fill out missing information from the data acquisition process.
To implement the time dimension of the modelling, data from different time instances have been collected.
It is clear that for content referring to past periods, only RGB image data are available. These data are
collected through developed search engine, retrieving images over distributed databases and the social
media. More specifically, for the Calw case our data go back to 1850s and 1860s, when the first glass-
plate photographs could be collected.
In order to understand and classify spatio-temporal regions in historic urban cities, deep learning
strategies are employed. The term deep learning refer to a variety of techniques that can understand
complicated patterns by automatically extracting the appropriate features for the task. In the context of 4D-
CH-Word, we have focused on convolutional neural networks (CNNs) for two different scenarios; a) timber
house identification and b) urban development monitoring. The deep learning approaches supported the
4D reconstruction; i.e. understanding the time period of the reconstructed areas by either, separating new
from old buildings, or newly developed areas. In the end, deep learning and advanced filtering approaches
allowed the creation of image clusters according to the time period.
The 4D-CH-World outcomes are presented via 4D visualization toolkits. The visualization methods have
been developed in a Web-based interface. A user is able to freely navigate within the reconstructed
environment and interact with it. Texturized detailed models depict accurate the current place, providing
further information when requested by the user. Furthermore, for specific areas, user can experience the
evolution through the years; see the current monument, how it was the previous decades and view the raw
images that supported the 3D reconstruction. Finally, by selecting a year as a reference (e.g. 2000), we
can see (if available) the difference in a specific area just by activating the time filter. That way, the 4D-CH-
word team, successfully combined 3D models of a scene associated to different time instances,
developing a system for 4D reconstruction and visualization of the results.
The final research achievement of 4D-CH-Word deal with recommendation system able to filter content
according to both user preferences and visual attributes. The recommendation system empowers the user
to interact with the platform and selects CH objects of interest displayed to him/her through suitable

2 of 3
profiles. Users’ profiles are created by measuring the feedback of the users to the system through the
activation of recursive machine learning algorithms that on-line update the system response to fit user’s
preferences.
The produced software was designed for tablets and phones. It is also possible to run the platform in a
standard desktop PC. However, touch screen is strongly suggested. The user is able to navigate freely to
the 3D world either as pedestrian (road view) either from above. At specific areas, indicated on the map,
we are able to see the actual image that led to the creation of the 3D object.
The tools developed are currently exploited for the digitalization of intangible cultural heritage such as
dances through a collaboration with the TERPSICHORE and iTreasures EU funded projects. The
innovative tools are also exploited under the framework of the international project being currently in
progress regarding the restoration of the Holy Aedicule of the Holy Sepulcher Church in Jerusalem.

Last update: 25 July 2018

Record number: 237889

Permalink: https://cordis.europa.eu/project/id/324523/reporting

© European Union, 2022

3 of 3