The generation of high-resolution orthoimages based on TLS data and close-range images – the case study

: The development of high-resolution geometric documentation plays a fundamental role in the conservation works and managing cultural heritage objects. Nowadays, orthoimages are increasingly being used for these purposes, as it is a combination of geometric accuracy (derived, among other things, from terrestrial laser scanning data and/or photogrammetric methods) and visual quality (based on information from images). The objective of this article is present the methodology of orthoimages generation based on the integration of data acquired by the Z+F 5006h terrestrial laser scanner and the Canon EOS 5D Mark II digital camera. In this investigation, the methodology of high-resolution and high-quality orthoimages was presented. The modified version of the Structure-from-Motion processed the data, and MultiView Stereo approaches based on integrating TLS data with close-range images and extended data analysis. The primary objective of the performed works presented in this paper was to optimise the quality of the high-resolution orthoimages.


Introduction
Modern non-invasive measurement technologies are commonly utilised to monitor, protect, manage and preserve cultural heritage. It is, therefore, essential to generate high-resolution architectural documentation in the form of 3D point clouds, 3D models, orthoimages and vector drawings [1][2][3][4][5][6]. For that purpose, image-based (passive methods, i.e. close-range photogrammetry) and range-based methods are used (i.e. Terrestrial Laser Scanning or Microwave) [5][6][7][8][9][10][11]. Both methods are burdened by errors (caused by the measurement system and the properties of the objects), which negative impact measurement accuracy and consequently contributes to the accuracy of the final architectural documentation. The main advantages and disadvantages of image and range-based method were described in many publications [i.e. [12][13][14][15][16], but the integration of close-range photogrammetric methods with terrestrial laser scanning increases the quality of generated documentation with the elimination of the disadvantages mentioned above [12][13][14]. A summary of the potential and limitations of image and range-based methods is presented in Table 1. Table 1. The comparison of the advantages and disadvantages of close-range photogrammetry and Terrestrial Laser Scanning [15,16].

Characteristic
Photogrammetry (image-based) From the point of view of cultural heritage researchers, one of the most valuable products of close-range photogrammetry is orthoimage. Ordinary digital images are acquired in a central projection, so making a direct measurement on the images is impossible. An orthoimage represents an object in an orthogonal projection; thus, orthoimages are products that combine metric and radiometric properties of photographs, taking into account the actual distances between elements in correct topology and geometrical relationship [4,17]. High-resolution orthoimages are created from the integration of products from image and range-based methods [4,9,18,19]. It is crucial for orthoimages of cultural heritage objects that the elements have to be accurately represented with attention to the smallest detail in order to achieve a high geometric and radiometric resolution of the final product.
This article aims to present the extended method of high-resolution orthoimages generation of decorated baroque interiors. In this investigation, the following issues are discussed: (1) the method of data acquisition and processing, (2) the method of quality assessment for TLS data and close image integration for high-resolution orthoimages generation, and (3) the comparison of quality of dense point cloud obtained from image and range-based techniquethe influence on the final Digital Surface Model (DSOM).

The test site description
The subject of the analysis was one of the large baroque, irregularly shaped rooms in the Museum of King Jan III's Palace at Wilanów. On the walls are gold patterned satin fabrics with a high sheen, and the decorative pattern is repeated throughout the material (Fig. 1a). The main entrance is on the east wall, and the south wall, there is a second entrance into the room. The west wall is divided into two diagonal wings and a central section with an alcove. On the left wing, there is a marble fireplace (Fig. 1b). On the right wing is another doorway leading to an unobstructed corridor. The door is difficult to see as it is upholstered in fabric ( Fig.1 c). On the north side is a window overlooking the garden (Fig. 1c).

The overview of the approach
The proposed method of orthoimages generation based on Terrestrial Laser Scanning and closerange images is a multi-stage process; it consists of (I) data acquisition and photogrammetric network design, (II) data pre-processing, (III) the analysis of the quality of pre-processing step and (IV) final orthoimage generation. Figure 2 presents the diagram of the research work and experiments performed. The idea of the orthoimages generation presented in this paper (according to the diagram shown in Figure 2) consists of the following stages:

Data acquisition and photogrammetric network design
The first stage of data acquisition was the distribution of photogrammetric control and check points in the form of black and white crosses (Fig. 3a), which in a later stage were used for the orientation of the close-range images, the TLS point clouds registration and the relative orientation of both datasets. Under the supervision of the conservator, a museum employee used special adhesive tape that did not damage the walls or cause paint fragments to detach; the control and check points were distributed evenly throughout the room. On each wall, 9/10 points were placed in such a way that none of them was in a straight vertical or horizontal line to the other and that the points were not concentrated in one place but were evenly distributed on the walls (Fig. 3b). A total of 43 points were distributed throughout the room. Each wall is covered in the central part with a decorative fabric on which it was impossible to place the points concerned, damaging it, so all the points were placed on white wooden wall elements surrounding the fabric section and on the plain wall elements, under the ceiling. Once the measurements were completed, all the marks were removed without damaging the parts of the walls on which they were deployed. The choice of locations for the terrestrial laser scanner was guided primarily by avoiding the occurrence of so-called blind spots so that all scans covered the entire room. The scanner was also positioned at different heights during the measurement using a tripod, which allowed it to eliminate the influence of an acute scanning angle. The scanner used for this investigation was a Z+F model Imager 5006h, with a minimum scanning range of 0.4 m. The measurement took place indoors, so the scanner's distance from the object's walls was no greater than 5 m. Therefore, the measurement accuracy ranges from 1.2 mm RMS on black surfaces to 0.4 mm RMS on white surfaces (Fig. 4a).
(a) (b) A total of nine scans of the 'Lady's Bedroom' room were taken at a scanning resolution of 3 mm/10 m. During the measurement, the room was lowered so as not to interfere with the scanner or generate noise. A sketch of the layout of the scanner stations is shown in Figure 4b.
The images were taken with Canon EOS 5D MARK II in horizontal strips with a frontal overlap of approximately 80% and a side overlap of approximately 60%. Difficulties in taking the photographs were caused by the gold fabric on the walls, on which the light scattered and reflected. The shutters were closed to negate the influence of sunlight, but the lighting was still not uniform. Illuminated stripes on the gold satin in irregular shades were forming. The first attempt to eliminate this negative phenomenon was to use a camera flash on a tripod, but the light it produced was insufficient, so two reflectors Profoto B10 Plus and two softboxes with umbrellas were used).
The reflectors were placed parallel to the wall currently being photographed at a distance of 1.5 m and their power was set to the highest value of 10. The height of their tripods was established according to the horizontal strip of wall currently being photographed. Taking into account the need for coverage between adjacent strips, photographs were taken from floor to ceiling. With each successive strip, the position of the reflectors was changed -they were raised so that their light fell on the corresponding strip of wall and uniformly illuminated it in the same way as on the previous strip. The first Photograph of each row was taken with a colour pattern to facilitate subsequent radiometric correction of the photographs and to eliminate differences in lighting. Approximately 11 rows of images were taken on each wall. The alcove walls in the depth of the room were photographed separately in the same way. The photographs for each row were taken perpendicular to the wall as far as possible, avoiding taking them at an angle. For the highest parts of the room, a ladder had to be used and the spotlights "pulled" to the highest level of the tripod to get the desired image.

Data pre-processing
Data pre-processing was carried out separately for point clouds acquired from the terrestrial laser scanning and close-range images. In the case of TLS data, the data pre-processing was a multi-step process in which: • automatic measurement signalised control points. For this, the dedicated Z+F LaserControl software was used [21], • point registration based on previously measured points using the target-based method [22,23] • filtering of points according to the intensity of the reflected laser beam, whereby it was possible to eliminate all points on which reflection or scattering of the laser beam occurred; a fragment of objects on which refraction of the beam occurs at the edges (so-called mixededge effect), etc. [5,[24][25][26] • on each of the scans, test areas were chosen and exported to .las format for further quality assessment The pre-processing of the close-range images was divided into the following stages: • images division into four chunks, and as a criterion for division, the distance of images from a specific wall was used. This made it possible to independently determine the camera's internal orientation elements for a specific group of images for each wall. Correct determination of the camera's internal orientation elements influences the accuracy of the image orientation and the generation of dense point clouds, which affect the final accuracy of the resulting orthoimages. • preliminary relative orientation of a group of images (chunk). The Agisoft Metashape software [27], which uses the Structure-from-Motion (SfM) method [28,29], was used for the relative orientation of the images. Carrying out this step makes it possible to check the correctness of the structure of the photogrammetric block of close-range images, • control/check points measurement and exterior orientation. The final image orientation stage involved the images' external orientation based on the points used to orient the TLS data. As with the previous step, Agisoft Metashape software was utilised using the SfM approach. • chunck relative orientation and dense point cloud generation. The final stage of orienting the images involves combining all the chunks into a single unit based on the designated external orientation elements of the images in the previous stage. The images thus oriented were used to generate a dense point cloud using the MultiView Stereo [30,31] method and the Semi-Global Matching approach [32]. As with the point clouds from TLS, the dense point cloud from the close-range images was exported to las format.

Point cloud analysis
The point clouds obtained from the two methods were analysed to asset the geometrical quality of point cloud registration. As a reference, the wooden parts of the wall and the fabric were chosen, and distances between point clouds were calculated. The second investigation was based on the determination of which dataset is suitable for shape reconstruction and Digital Surface Model Generation. The test required a plane to be fitted into each of the selected fragments of point clouds, namely fabrics and wooden elements, and then the distance between the fragments and the fitted plane was calculated. All analyses were carried out in CloudCompare software.

Data integration and final orthoimage generation
The final processing step involved integrating TLS data and point clouds from the MVS method and generating orthoimages. The data integration generated the point cloud used to generate the Digital Surface Object Model to orthorectify the images. For this purpose, the results of the previous analyses were used, and the integrated point clouds were treated holistically as a single dataset. Orthoimage generation is a well-known process in close-range photogrammetry [33], and consists of the following steps: (1) generation of DSOMs in a TIN structure, (2) Orthorectification of individual images to change from a central projection to an orthogonal projection, (3) Mosaicking and tonal alignment to combine orthorectified image fragments into a single metric product.

Data pre-processing
For the registration process of the point clouds acquired from the different TLS positions, 43 control points were used. A total of 283 points were measured on 9 scans. The registration resulted in the following accuracies: linear RMSE of 1.3 mm, a linear average deviation of 0.6 mm and a maximum linear deviation of 3.6 mm. In the next step, point clouds were filtered based on the intensity thresholds lower than 0.5% and higher than 99% of the reflectance. As a result, most of the golden satin fabric on the walls was removed from the point clouds (Fig. 5a). Also removed were the points that were outside the study area, which were recorded through a window located on the north wall of the room and the floor. Data was acquired for the entire room during the measurements, but each wall had to be processed separately. In this investigation, only the example of the one wall was presented. A total of 206 images of the selected wall were taken, divided into 11 strips (separate chunks) and processed, related to the height of acquired images. In each chunk, photographs were firstly relatively oriented. All chunks were relatively oriented, and the characteristic points on the fabric were chosen as a tie points. Finally, images were oriented in an assumed reference system based on the same points used for TLS point cloud registration. As a result of the orientation of the images, the RMSE at the control points was for the X -1.4 mm, Y -3.8 mm and Z -1.8 mm axes and for the check points for the X -2.2 mm, Y-2.2 mm and Z -2.1 mm axes, which proves that the image orientation process was carried out correctly. A dense point cloud was generated in the last data pre-processing step (Fig. 5b).

Point cloud analysis
To assess the deviations between the TLS point clouds, data was selected from site 1, from which the section of wall under development was measured at the lowest possible angle of relative normal to the surface. The distances between the selected point clouds were analysed in CloudCompare software using the Compute cloud/cloud distance tool, and the linear distance value was investigated. The results are presented in the form of histograms (Fig 6) and statistical analysis (average deviation and standard deviation, Table 2). An analysis of the histograms (Fig. 6) and statistics presented in Table 2 shows that the deviations of the most significant values are in the range of 0.4-2.0 mm. Differences can only be seen in the number of counts. The presented investigation results show that the TLS point clouds registration process was carried out correctly, but only for scan pairs 1 and 3; the mean deviations between the point clouds are significantly different than for the other pairs. The point cloud from scan position 3 should not be used for further studies. The second stage of the geometric quality analysis of the point clouds involved analysing the accuracy of mapping the decorative satin fabrics based on the point cloud deviations from the reference plane. In the literature, examples of shape misrepresentation on this type of object can be found using TLS [34]. This is a crucial step in the generation of orthoimages, as it significantly affects the quality of the Digital Object Model that is the basis for image orthorectification. Table 3 shows the results of the performed investigations. The analyses presented in Tabel. 3 showed that the lowest standard deviations of the plane fit were obtained for the point cloud from the photographs and the point cloud acquired from TLS position 2.
The final assessment of the quality of the point clouds consisted of evaluating the completeness of the wooden shapes and materials representation using the each dual technique separately (Tab. 4). After analysing all the data, it was decided that the white wall elements would be taken from point clouds from terrestrial laser scanning and the fabric from the point cloud from the photographs. The decision was made based on a comparison, where it was noticed that there was a significant loss in the structure of the white wooden wall elements on the cloud generated from the images. On the clouds from the scans, data gaps are visible where the fabric is present because the laser beam has of the laser has scattered or reflected off its shiny surface

Summary
One of the most important goals in the investigation of cultural heritage objects is to achieve high accuracy in the final documentation. Unfortunately, due to the measurement limitation of image and range-based sensors, it is necessary to perform data integration. Applied separately TLS point cloud in cultural heritage object investigation is limited to properly determining the decorative fabrics. Using the SfM technique combined with MVS did not allow correct shape determination of flat surfaces such as decorative wooden parts. Integrating both techniques allows for a high-resolution Digital Surface Object Model (Fig. 7) and final orthoimage (Fig. 8) generation.  The resulting orthoimage is characterised by high quality (final Ground Sampling distance is equal to 0.5 mm) and high accuracy, both for fabrics and simple elements. The use of only a specific part of the point cloud allowed the elimination of the negative impact of data integration on the correctness of the projection of elements of uniform texture. The integration of datasets allows for the generation of a photogrammetric product that meets the accuracy requirements of the investigation of historical objects.
In summary, it may be stated that integrating Terrestrial Laser Scanning data with point clouds from close-range images dense matching allows for reducing the limitation of each sensor and improving the geometrical quality of final documentation. Future work can involve optimising the automatic process of detecting parts that might be used in the Digital Surface Object Model generation and automatic dataset co-registration without photogrammetric network determination.
Authors Contributions: J.M. and A.R. organised the conceptualisation of the idea and the methodology employed in this paper. Following that, J.M and A.R. carried out the experimental design. A.R worked on the data acquisition at the Museum of King Jan III's Palace at Wilanów. J.M. carried out the original writing and