industry news
Subscribe Now

A Novel Neural Network for Preserving Cultural Heritage via 3D Image Reconstruction

Researchers develop a novel neural network model that effectively reconstructs 3D digital images of relief-type cultural heritage objects from old photos
Relief-type cultural heritage objects are commonly found in many historical sites worldwide, but often suffer from varying levels of damage and deterioration. Traditional methods for image reconstruction require extensive manual labor and specialized knowledge. Now, researchers have developed a novel neural network model that can reconstruct these reliefs as three-dimensional digital images from old photographs containing their pre-damage information. This innovative technology paves the way for accurate digital preservation of valuable cultural heritage objects.
Relief carvings or relief sculptures are cultural heritage objects with figures that protrude from a background such as a wall or slab, creating a sense of depth. Commonly found at historical sites worldwide, these artworks are considered to be of immense historical and cultural value. Unfortunately, many such relief carvings at heritage sites across the world suffer from varying degrees of damage and deterioration over time. While modern 3D scanning and photogrammetry techniques can digitally preserve their current form, they cannot restore the original appearance of these carvings before damage. Additionally, traditional methods for restoring them are laborious, and require extensive manual intervention and specialized knowledge.
One promising method is 3D digital reconstruction of these reliefs from old photos, captured before the damage or deterioration. Unlike 3D sculptures or 2D paintings, reliefs have a shallow depth and are meant to be viewed from the front or either side. This means that a single image can effectively provide most of the information required for 3D digital reconstruction algorithms.
In a breakthrough, a multinational research team led by Professor Satoshi Tanaka from the College of Information Science and Engineering at Ritsumeikan University, Japan, along with Dr. Jiao Pan, from the University of Science and Technology Beijing, China, developed an innovative multi-task neural network for 3D reconstruction and digital preservation of reliefs using old photos. “Previously, we proposed a 3D reconstruction method for old reliefs based on monocular depth estimation from photos. Although we achieved 95% reconstruction accuracy, finer details such as human faces and decorations were still missing. This was due to the high compression of depth values in 2D relief images, making it difficult to extract depth variations along edges. Our new method tackles this by enhancing depth estimation, particularly along soft edges, using a novel edge-detection approach,” explains Prof. Tanaka.
The team also included Prof. Liang Li from Ritsumeikan University and Prof. Xiaojuan Ban from the University of Science and Technology Beijing. Their study was presented orally at the international conference ACM Multimedia 2024, held in Australia in October, and was published in MM ’24: Proceedings of the 32nd ACM International Conference on Multimedia on October 28, 2024.
The proposed multi-task neural network performs three tasks, namely semantic segmentation, depth estimation, and soft-edge detection, which work together to enhance the accuracy of 3D reconstruction. The core strength of the network lies in its depth estimation, achieved through a novel soft-edge detector and an edge matching module. Unlike the conventional binary edge classification, the soft-edge detector treats edge detection of relief data as a multi-classification task. Edges in relief images not only represent changes in brightness but also variations in curvature, known as “soft edges”. The soft-edge detector determines the degree of “softness” of these edges in relief images, enhancing depth estimation.
The edge matching module comprises two soft-edge detectors that extract multi-class soft-edge maps and a depth map, from an input relief photo. By matching and detecting differences between the two maps, the network focuses more on the soft-edge regions, resulting in more detailed depth estimation. Finally, the network optimizes a dynamic edge-enhanced loss function, which includes loss from all three tasks, and produces clear and detailed 3D images of reliefs.
The researchers applied this innovative model to reconstruct the hidden reliefs of Borobudur Temple. “The wall reliefs on the ground level of the Borobudur Temple, a UNESCO World Heritage Site in Indonesia, are covered by stone walls due to reinforcement work carried out during the Dutch colonial period and cannot be viewed. Our multi-task neural network successfully reconstructed these hidden sections of Borobudur’s ground-level reliefs from surviving old photographs. Through computer visualization and virtual reality, our research now allows virtual exploration of these unseen treasures,” says Prof. Tanaka, highlighting the potential impact of their work.
Discussing the future implications of these findings, he says, “Our technology holds vast potential for preserving and sharing cultural heritage. It opens new opportunities not only for archeologists but also for immersive virtual experiences through VR and metaverse technologies, preserving global heritage for future generations.
 
Reference
Title of original paper: Reconstructing, Understanding, and Analyzing Relief-Type Cultural Heritage from a Single Old Photo
Conference: ACM Multimedia 2024

Leave a Reply

featured blogs
Dec 2, 2024
The Wi-SUN Smart City Living Lab Challenge names the winners with Farmer's Voice, a voice command app for agriculture use, taking first place. Read the blog....
Dec 3, 2024
I've just seen something that is totally droolworthy, which may explain why I'm currently drooling all over my keyboard....

Libby's Lab

Libby's Lab - Scopes Out Silicon Labs EFRxG22 Development Tools

Sponsored by Mouser Electronics and Silicon Labs

Join Libby in this episode of “Libby’s Lab” as she explores the Silicon Labs EFR32xG22 Development Tools, available at Mouser.com! These versatile tools are perfect for engineers developing wireless applications with Bluetooth®, Zigbee®, or proprietary protocols. Designed for energy efficiency and ease of use, the starter kit simplifies development for IoT, smart home, and industrial devices. From low-power IoT projects to fitness trackers and medical devices, these tools offer multi-protocol support, reliable performance, and hassle-free setup. Watch as Libby and Demo dive into how these tools can bring wireless projects to life. Keep your circuits charged and your ideas sparking!

Click here for more information about Silicon Labs xG22 Development Tools

featured paper

Quantized Neural Networks for FPGA Inference

Sponsored by Intel

Implementing a low precision network in FPGA hardware for efficient inferencing provides numerous advantages when it comes to meeting demanding specifications. The increased flexibility allows optimization of throughput, overall power consumption, resource usage, device size, TOPs/watt, and deterministic latency. These are important benefits where scaling and efficiency are inherent requirements of the application.

Click to read more

featured chalk talk

Ultra-low Power Fuel Gauging for Rechargeable Embedded Devices
Fuel gauging is a critical component of today’s rechargeable embedded devices. In this episode of Chalk Talk, Amelia Dalton and Robin Saltnes of Nordic Semiconductor explore the variety of benefits that Nordic Semiconductor’s nPM1300 PMIC brings to rechargeable embedded devices, the details of the fuel gauge system at the heart of this solution, and the five easy steps that you can take to implement this solution into your next embedded design.
May 8, 2024
39,102 views