Georeferencing Accuracy Assessment of Historical Aerial Photos Using a Custom-Built Online Georeferencing Tool

Abstract

As one of the earliest forms of remote sensing, aerial photography has been regarded as an important part of the mapmaking process. Aerial photos, especially historical aerial photos, provide significant amount of valuable information for many applications and fields. However, due to limited funding support, most historical aerial photos have not been digitized and georeferenced yet, which substantially limits their utility for today’s computer-based image processing and analysis. Traditionally, historical aerial photos are georeferenced with desktop GIS software applications. However, this method is expensive, time-consuming, and labor-intensive. To address these limitations, this research developed a custom-built online georeferencing tool to enable georeferencing digitized historical aerial photos in a web environment, which is able to georeference historical aerial photos in a rapid and cost-effective manner. To evaluate the georeferencing performance, a set of 50 historical aerial photos were georeferenced with not only the developed online georeferencing tool but also two commercial desktop software programs. Research results revealed the custom-built online georeferencing tool provided the highest degree of accuracy while maximizing its accessibility.

1. Introduction

The oldest form of remote sensing, aerial photography, is the taking of photos in the air with a mounted or handheld camera on airborne platforms such as manned aircraft, unmanned aerial vehicles (UAVs), rockets, balloons, blimps, kites, parachutes, and even pigeons. Aerial photography has been developing since 1858 when the first known aerial photo was taken on a balloon by a French photographer and balloonist Gaspar Felix Tournachon [1]. Due to its pictorial nature, an aerial photo can provide a straightforward depiction of the landscape a specific area at a given time [1].

Based on data collection time, aerial photos can be grouped as historical aerial photos and modern aerial photos. Noting that the definition of historical and contemporary benchmark is dynamic, and based on collection times, aerial photos collected prior to 1990s are defined as historical aerial photos, and photos collected from 1990s to present are defined as contemporary aerial photos in context of this study. In addition, historical aerial photos are primarily collected with film cameras, while contemporary aerial photos are primarily collected with digital cameras.

Aerial photography provides a means to observe the Earth “as the birds do” [2]. Unlike textual documents, aerial photos can document Earth’s surface without topical selection (i.e., purposeful omission of the landscape being photographed) and human interpretation (i.e., personal perspectives on the landscape being photographed). In early days, aerial photography had a limited usage due to high cost and risk. Since the early 1900s, the utility of aerial photography has been expanded remarkably with the introduction of powered flight [2]. Aerial photos contain rich and synoptic information about the Earth’s surface. While satellite imagery is another common source of information about the Earth’s surface, its utility for Earth’s surface long-term monitoring is severely constrained by its limited records (e.g., the first set of imagery was collected in 1972). In contrast, aerial photographs provide the longest-available, temporally continuous, and spatially complete record of Earth’s surface, dating from the early 1930s in some cases [3]. Thus, aerial photos are ideal data sources for mapping Earth’s surface and monitoring its changes at a local, regional, and even national scale. They have been widely used in historical and cultural studies [4,5], archaeology [6,7], socioecology [8], landscape studies [9,10,11], civil engineering [12], geography [13], environmental studies [14], ecological studies [3], land use planning [15], forest studies [16], and property boundary disputes [17].

Aerial photos have been routinely collected in the U.S. since the 1930s [18]. Some of the popular national programs include the National High Altitude Photography (NHAP) program, the National Agricultural Imagery Program (NAIP), and the National Aerial Photography Program (NAPP). Other programs have also been developed by states, counties, or regional metropolitan planning organizations (MPO) such as the Mid-Region Council of Governments in New Mexico. Most of the programs collect aerial photos on an annual basis or a biennial basis, and the collection is still ongoing due to high demand.

As a practical source of information, historical aerial photos data are valuable because they can be used to reconstruct the chronology of changes in the land [19]. This is because historical aerial photos can document the landscape of a large area at the time the photos were taken, which works as a landscape snapshot, and a time series of historical aerial photos of a specific area can be used to extract landscape changes over time. However, historical aerial photos also have inherent weaknesses. The major disadvantage of historical aerial photos is that all historical photography is film-based. Therefore, in order to make historical aerial photos more useful or serviceable for different purposes [20], it is necessary to convert them from film or printed hardcopy to a digital format (digitization), and subsequently, it is necessary to georeference them (i.e., assigning ground coordinates to each pixel of a historical aerial photo to define its location on the Earth). Traditionally, historical aerial photos are georeferenced with desktop geographic information system (GIS) or remote sensing software applications. However, desktop-based georeferencing method is expensive, time-consuming, and labor-intensive [21]. Currently, most of the historical aerial photos are still un-georeferenced, and therefore, it is necessary to develop a solution to increase georeferencing productivity at a low cost.

Recent advancements in web-based applications, especially free and open source software for geospatial applications or known as FOSS4G, provide a potential method for georeferencing. To make it distinct from traditional desktop-based georeferencing, the web-based georeferencing is termed online georeferencing. Based on our review of literature, the use of online georeferencing technique for georeferencing historical aerial photos is lacking and it presents a significant gap in the research. With funding from the National Historical Publications and Records Commission (NHPRC) of the U.S. National Archives and Records Administration (NARA), a custom-built web application for georeferencing was developed to address the aforementioned limitations of desktop software programs. This study was focused on collecting data about georeferencing accuracy of the custom-built online tool, and subsequently, conducting empirical analysis to investigate if the custom-built online georeferencing web application outperforms traditional desktop software programs that are commonly used by geospatial practitioners.

2. Background

Historically, film cameras were primarily used for aerial photography, while digital cameras have only relatively recently become mainstream devices since late 1990s. Aerial photos collected by digital cameras are in softcopy format (i.e., digital format) and geotagged, and therefore, they can be used directly on computers without georeferencing operations [1]. In contrast, aerial photos collected by film cameras are stored in film format or printed hardcopy format. Prior to conducting georeferencing with computers, they need to be digitized (i.e., scanned) with a photogrammetric scanner at a pre-determined resolution (i.e., dots per inch or dpi) [1].

Georeferencing is the process of assigning absolute ground coordinates to geographic features within a spatial reference system [1]. In context of this study, georeferencing transforms a scanned historical aerial photo so that it appears in its actual location on the Earth’s surface. It should be noted that georeferencing is different from orthocorrection. Orthocorrection not only relates geographic features to absolute ground coordinates but also reduces distortions due to topographic variations [1].

Currently, the most widely adopted method to georeference historical aerial photos is through the use of desktop GIS or remote sensing software programs, including, but not limited to, ArcGIS (commercial GIS software), ERDAS (commercial remote sensing software), ENVI (commercial remote sensing software), and QGIS (free and open source software). These software programs may have different graphic user interfaces (GUIs), but end to end workflows are fundamentally similar. Users need to upload a digital (e.g., digitized), un-georeferenced image (e.g., a historical aerial photo) to a desktop software program, assign ground control points (GCPs) to connect the un-georeferenced image to known locations on the reference map, review the errors associated with GCPs, relocate GCPs to reduce errors to an acceptable level, conduct image coordinate transformation, and export the georeferenced image. In context of this study, a GCP is defined as a location with known positions (e.g., longitude, latitude, and elevation) on the surface of the Earth, which can be used to georeference historical aerial photos.

Adequate and non-correlated (evenly distributed across the un-georeferenced image) GCPs are used in image coordinate transformation to achieve accurate results. Image coordinate transformation is an important process to warp and shift a un-georeferenced image to match the coordinates of the reference image with acceptable errors. The selection of adequate GCPs is dependent on the complexity of the image transformation, but this does not indicate that more GCPs will yield a georeferenced image with higher accuracy. Sometimes more GCPs will lead to inaccurate results, especially when all the GCPs are concentrated in one area. Many models are available to perform coordinate transformation, including polynomial, rubber sheeting, and spline functions [22]. However, polynomial warping is the most commonly used transformation model due to its computational efficiency and resilience to outliers [23]. For first, second and third order polynomial transformation, three, six, and ten GCPs are minimally needed, respectively [1].

In practice, the minimum number of GCPs recommended is at least twice the aforementioned number because only GCPs beyond this minimum number are used as checkpoints to evaluate model fit [23]. However, operationally it is always challenging to get adequate checkpoints because they need to be at least three times more accurate than the tested product [24]. Oftentimes ground surveys with high-accuracy measurement equipment such as real-time kinematic (RTK) are needed to establish checkpoints. Root-mean-squared error (RMSE), which is the square root of the mean of the square of all the residuals (i.e., the difference between the observed position of a GCP on the georeferenced image and the expected position of the same GCP on the reference image), is a commonly used measure of georeferencing accuracy [1]. The American Society for Photogrammetry and Remote Sensing (ASPRS) Manual of Photogrammetry recommends that a georeferenced aerial photo have a RMSE less than one to three pixels [25], but accuracy standards should be driven by the intended use of the data being georeferenced [23].

Although widely adopted, desktop software programs for georeferencing have many drawbacks. Firstly, these desktop software programs have deep learning curves because they are designed to perform all-inclusive tasks, including not only visualization and spatial analysis but also image processing and mapping. Imagery (e.g., historical aerial photos) or map georeferencing is only one of the tasks. Although there are many options, normally users select a software program for georeferencing based on their familiarity. Therefore, productivity may be severely impacted if users have to change their selected software programs during a project. Secondly, these software programs are not designed to georeference loads of aerial photos that were collected during a specific aerial survey. Oftentimes users will use their selected desktop software program to georeference one aerial photo, export it, and then go to the next. This is not operationally practical for multiusers who are collaboratively georeferencing a collection of historical aerial photos. In the worst-case scenario, each user selects a distinctive software program to conduct georeferencing, causing inconsistencies in quality and format. Lastly, as their type indicates, these software programs are desktop applications that need to be run locally on a computer device. They must be installed on a computer, and strict hardware requirements need to be met to ensure they function correctly as intended. Their performance will be severely restricted by hardware, which will ultimately limit productivity.

In recent years, web-based applications have become increasingly popular with GIS and remote sensing users (e.g., internet mapping or web mapping). When compared with desktop applications, web applications have many benefits, including, but not limited to, great accessibility across devices, shallow learning curve, no need for installation, easy maintenance, automatic updates, capable of using resources available over the internet, capable of delivering complex applications from a centralized cyberinfrastructure, and capable of providing more interactive and media-rich users interfaces [26].

The advancements of web-based applications have enabled the development of online georeferencing tools. The workflow of using an online georeferencing tool is very similar to that of a desktop-based georeferencing tool, but instead of conducting georeferencing with a desktop application, online-georeferencing only requires a web browser and internet access [21]. Online georeferencing has many benefits. First, online georeferencing has a shallow learning curve since it is designed only for georeferencing operations. Users can become proficient in georeferencing within a short amount of time. Second, substantial amounts of images can be georeferenced with limited resources (i.e., internal personnel, hardware, and software) because online georeferencing enables crowdsourcing. Desktop-based georeferencing requires specialized staff, while online georeferencing allows specialists, professionals, teachers, students, and the public to participate in georeferencing concurrently, which can substantially increase production. Third, online georeferencing enables users to process images from anywhere around the world without any need of special software as long as they have internet access and a web browser, which is particularly important because many workers may need to work from home or work remotely due to various restrictions. Last, online georeferencing does not have a high demand for computing power since it only requires a modern web browser and internet access, which greatly reduces hardware requirements.

In recent years, a few studies have explored the utility of online georeferencing. Kowal and Pridal used an online georeferencer application developed by Klokan Technologies to georeference historical scanned maps, and they concluded that online georeferencing could benefit cartographic research, public engagement, and improved data management [27]. Arias-Carrasco et al. developed a web application named OUTBREAK to georeference the location of disease outbreak cases, concluding that that online georeferencing held the potential to visualize cases at city and even neighborhood levels [28]. Ellwood et al. also developed an online tool to georeference biodiversity research specimens, which substantially engaged public participation [29]. A review of the literature reveals that an online georeferencing tool for historical aerial photos is lacking, and therefore, this study was focused on developing such a tool and evaluating its performance.

3. Materials and Methods

3.1. Online Georeferencing Tool Development

As mentioned in Section 1, with funding support from the NHPRC of NARA, a web application was developed to address desktop application’s limitations for georeferencing. The developed web application can be used to georeference, document, and web-publish digitized historical aerial photos. That being said, the developed web application can serve not only as an online georeferencing tool but also as a data clearinghouse tool. More specifically, the web application has been developed to enable: (1) georeferencing historical aerial photos online; (2) using a bounding box to represent the ground coverage extent of a historical aerial photo; and (3) conducting spatial search to increase of the discoverability and usage of historical aerial photos.

In addition, the developed web application can be used by anyone who is interested in contributing to historical aerial photo collections at no cost. Users can start using the online georeferencing tool as soon as they sign up their free accounts. Users can also login with their Google accounts or their Open Researcher and Contributor Identifier (ORCID) accounts as part of the OAuth authorization process. OAuth is an open standard for access delegation, and it is commonly used by web users to grant website or web application access to their information on other websites without creating an account or registering. The implementation of OAuth permits a broad range of possible users.

The aforementioned web application was developed based on a set of open standards and open source software and libraries, including, but not limited to, Open Geospatial Consortium (OGC) standards, World Wide Web Consortium (W3C) standards, and Open Source Geospatial Foundation (OSGeo) Geospatial Data Abstraction Library (GDAL). Application Programming Interfaces (APIs) for the web application were also developed and have been freely shared with archival and record management organizations across the United States to enable them to develop similar web applications, and ultimately, promoting the accessibility and use of historical aerial photos to encourage understanding of our nature, history, culture, natural and culture heritage, and geography.

It is worth noting that the developed web application uses the Geographic Storage, Transformation, and Retrieval Engine (GSToRE) as the backend catalog of the georeferenced historical aerial photos ready for web-publication. GSToRE, which is a flexible and scalable data management, discovery, access, and delivery platform that supports open standards (e.g., open geospatial consortium or OGC standards) for client discovery and access, has been developed by EDAC at the University of New Mexico since 2009 [30]. GSToRE is built upon the principle of a service-oriented architecture that provides a layer of abstraction between data as well as metadata management technologies and the client applications that consume platform published services [30]. Over the years, GSToRE has also been improved to incorporate a web service-based model that exposes data discovery, access, visualization, and administrative capabilities to client applications via various services including representational state transfer (REST) services, OGC web map services (WMS), OGC web feature services (WFS), and OGC web coverage services (WCS). Another unique feature of GSToRE is that it supports both spatial and non-spatial data.

GSToRE was developed with two Python frameworks, including Pyramid and Django. The databases used for GSToRE are PostgreSQL and PostGIS (i.e., canonical data model and spatial data processing), MongoDB (i.e., vector and tabular attribute storage), and ElasticSearch (i.e., search engine). GSToRE has been used by various statewide programs as the backend catalog to develop, implement, and operate a data clearinghouse. These projects include, but not limited to, New Mexico Resource Geographic Information System (NM RGIS) funded by the New Mexico Legislation, New Mexico Established Program to Stimulate Competitive Research (NM EPSCoR) funded by the National Science Foundation (NSF), and Data Observation Network for Earth (DataONE) funded by NSF [30]. GSToRE is also freely accessible to the public. Any users can download it from EDAC’s GitHub repository. The links for downloading GSToRE and accessing the developed web application and its APIs are provided in the Data Availability Statement.

The development process started with an analysis of the project requirements. The project requires multiple geospatial libraries to accomplish online georeferencing capabilities. GDAL was used to establish most geoprocessing capabilities within the web application. Python was selected for the ease of implementing these geoprocessing capabilities using the GDAL Python bindings. Flask was selected as the web framework as it offers a lightweight and flexible design. Figure 1 shows the development flowchart.

The initial steps of the development included the creation of full authentication capabilities and user landing pages with Hyper Text Markup Language (HTML) and JavaScript. Subsequently, the focus was then turned to establish a methodology that can track each historical aerial photo throughout the system. Upon completion of the local tracking system, a georeferencing user interface was developed with HTML and JavaScript to enable end users to provide GCPs. The display of a un-georeferenced image in a web map was accomplished with a custom WMS. A un-georeferenced image service was presented alongside a WMS reference image. The display of un-georeferenced image and reference image allows the end user to choose a pixel coordinate on the un-georeferenced image and a corresponding ground coordinate on the reference image, which are used as one pair of the matching GCPs. The GDAL libraries were used to create a georeferenced image and compute RMSE values when appropriate amount of GCPs presented.

The next step was to establish the Quality Assurance/Quality Control (QA/QC) and web-publication process. The QA/QC and web-publication process was developed to enable users to review and accept images to publish on an online catalog. For this study, we leveraged the catalog capabilities of the GSToRE application to allow rapid and stable public access to the web-published historical aerial photos. Noting that the georeferencing process does not require the GSToRE application to complete any of the georeferencing steps. That being said, GSToRE is only used as a final landing place that offers a host of services and capabilities. In addition, it should be noted that a MySQL database was used the backend database of the developed web application to store and manage data and metadata. The last step was to enable bounding box display and spatial search. For this study, we leveraged the bounding box display and spatial search capabilities of the GSToRE application to allow for the display of an image’s ground coverage and the search of images that are located within a user-defined area of interest (AOI). Figure 2 shows the architecture of the developed georeferencing web application. The workflow starts with a raw raster image and GCPs and ends with a georeferenced image.

3.2. Online Georeferencing Tool Capabilities

Online georeferencing is one of the primary spatial operations that have been enabled in the developed web application. One of the greatest advantages of online georeferencing is that it does not require installation on a computer, which decentralizes the georeferencing software. The most important advantage of software decentralization that we identified during the COVID-19 pandemic is that users can perform georeferencing operations from anywhere around the world through web browsers in the presence of an internet connection. That said, users can georeference digitized historical aerial photos in front of a computer at home without any need of special GIS software as long as they have a modern web browser and internet access that they can login with their registered accounts. Another important advantage of online georeferencing is that it is much faster than desktop georeferencing since it provides a simplified workflow for georeferencing, documenting, and web publishing historical aerial photos, and ultimately, improving georeferencing productivity with limited personnel, hardware, and software [21].

Traditionally, georeferencing, documenting, and web publishing are conducted in three steps. First, georeferencing is conducted on a computer with the aid of a GIS desktop application, which requires specialized staff on a regular basis. Afterward, metadata records are created by specialized staff on a computer with the aid of a metadata editing desktop application. At last, georeferenced historical aerial photos and associated metadata records are web published by developers via a data clearinghouse. With the developed online georeferencing tool, a lay user with a short training course will be able to georeference, document, and web publish a substantial amount of historical aerial photos across multiple projects. That being said, users do not use any specific desktop applications on a computer as all these operations can be conducted with a single web application. An additional benefit of integrating all operations is that users can transfer digitized and georeferenced historical aerial photos and their associated metadata records between different folders and file directories seamlessly without workflow interruptions.

The online georeferencing tool uses high spatial resolution mosaicked digital orthimages of the United States as a reference image to assign spatial coordinates to a historical aerial photo’s pixels. The orthoimages are provided by Google Maps API, and the majority of the United States has a 1 m spatial resolution (NAIP aerial images). However, many areas have a 6-inch or 0.1524 m spatial resolution, and certain areas even have a 3-inch or 0.0762 m spatial resolution. In addition, the tool has a graphical web interface that allows users to upload a digitized historical aerial photo (i.e., raw image), input and edit metadata information (the web application adds the information to the metadata record prior to web-publication), and place it alongside the reference image. By default, the raw image is being placed on the left side while the reference image is placed on the right side, but the placement could be rearranged. Moreover, from the same web interface, users are able to not only rotate a raw image if it is not oriented to the north but also browse (e.g., pan, zoom in, and zoom out, etc.) the raw image and the reference image. Furthermore, users can place corresponding GCPs on both the raw image and the reference image.

Once GCPs are placed and submitted, the tool will use GDAL to transform the raw image to its real world coordinates. GDAL provides a robust and dynamic means to determine the proper order of polynomial transformation based on the amount of input GCPs. First, second, and third order polynomial transformation will be used when more than three, six, and ten GCPs are presented, respectively. At last, with the online georeferencing tool, users can generate a georeferenced historical aerial photo from the transformed image and upload it and its accompanying metadata files to GSToRE for QA/QC review (e.g., RMSE < 3 pixels and complete metadata record). A georeferenced historical aerial photo will be web-published if it passes QA/QC review, and if not, it will be returned to the user for revision work and resubmission for review. Figure 3 shows the user interface of the online georeferencing tool. Users only need a web browser to use this tool. A total amount of 8 GCPs are placed on both the raw image and the reference image. The georeferencing accuracy is dynamically shown on the web interface.

The developed web application also enabled bounding box display (Figure 4). For this study, a bounding box is defined as a rectangular border that fully encloses a historical aerial photo. It is an effective means of showing the ground coverage of a historical geospatial product such as a historical aerial photo [31]. User can utilize this tool to explore if a digitized, georeferenced, and web-published historical aerial photo’s ground coverage is within a specific AOI prior to downloading it, which can substantially improve the efficiency of data search and download.

Additionally, the developed web application enabled spatial search (Figure 5), which can substantially improve the discoverability and use of historical aerial photo collections [21]. From the perspective of a data clearinghouse, an effective search operation should allow users to query data through not only textual search (e.g., keywords or location names) and catalog browsing (e.g., temporal search by catalog year) but also spatial search. In context of this research, spatial search is defined as custom query of historical aerial photos by drawing a rectangle search box. Historical aerial photos that are completely within the boundary of the rectangle search box will be returned for users to browse, select, and download. That said, historical aerial photos that are partially within or intersect the rectangle search will note be returned. This limitation is caused by the GDAL library that was used for developing the spatial search tool because it only supports completely within operation, not partially within or intersect operation.

3.3. New Mexico’s Historical Aerial Photo Collection

The State of New Mexico is rich in aerial photo acquisitions, primarily through state programs, federal and state partnered programs, or privately funded programs. A majority of New Mexico’s historical aerial photo collection is managed and maintained by the Earth Data Analysis Center (EDAC) at the University of New Mexico, which is an applied research and technical services center that serves the spatial needs of the federal, regional, state, local, tribal, educational, and private communities. EDAC was established at the University of New Mexico (UNM) in 1964 to transfer the National Aeronautics and Space Administration (NASA) space-based technology to the private and public sectors.

The New Mexico Archive (NM Archive) program was established in 1968 by EDAC. Historical aerial photo collections are the NM Archive’s primary source of interest, and holdings include historical aerial photos in paper (printed hardcopy), films (softcopy), and digital (electronic softcopy) format. Although the NM Archive program manages and maintains approximately 315,000 historical aerial photos, most of them remain un-digitized and un-georeferenced on aging film rolls which substantially limits their access and usage because the rapid adoption of digital technologies such as computers. The slow progress of digitization and georeferencing is due to the lack of funding support.

With funding support from NHPRC of NARA, 30,000 historical aerial photos have been scanned, georeferenced, documented, indexed, archived, and web-published. These 30,000 historical aerial photos were collected from 1934 to 1986 through a variety of programs. Noting that the majority of these processed historical aerial photos are located in the Grants Mineral Belt area in New Mexico, which was the primary area for uranium extraction and production activities in the United States from 1930s to 1990s (Figure 6). It is a key area of interest to conduct historical and cultural research, archaeological studies, and environmental studies, because health risks from uranium mining are substantial and ongoing, although most of the commercial mines and mills are no longer active.

3.4. Data Acquisition and Preprocessing

To complete this study, 50 photos were selected for georeferencing accuracy analysis from the aforementioned 30,000 historical aerial photo collection that covers the Grants Mineral Belt area. A set of 25 photos were selected from a roll of film containing imagery from the 1986 Western New Mexico Aerial Survey Project for the New Mexico State Engineer Office (SEO). The photos were acquired at a scale of 1:18,000 by Thomas R. Mann and Associates. A second set of 25 photo was selected from a roll of film acquired by Pacific Western Technology in 1972 at a scale of 1:30,000. All photos were selected over a wide range of terrain in Cibola County, NM to ensure a high degree of representativeness.

All 50 historical aerial photos had remained un-digitized and un-georeferenced on aging aerial film rolls which made them not readily available to users prior to project start. Aerial films are delicate and they require a range of special storage requirements to ensure top working condition over time. These requirements include, but not limited to, temperature, moisture, and dust condition. Fundamentally, aerial films are similar to photographic films that are commonly used in handheld film cameras. However, instead of using 35 mm format (small format), aerial films normally present in rolls that are 25.4 cm (10-inch) wide (large format) and range in length from 61 m (200 feet) to 152 m (500 feet).

Two types of aerial films have been widely used for aerial photography, including panchromatic films and natural color films. Panchromatic films are also known as black and white films and the aerial photos acquired with this type of film are printed as hardcopy gray-scale photos with a size of 25.4 by 25.4 cm or 10 by 10-inch (Figure 7). Natural color films are also known as true color films and the aerial photos acquired with this type of film are printed as hardcopy true color images with a size as gray-scale photos (Figure 8). It should be noted that most historical aerial photos were acquired with panchromatic films, although some of them were collected by natural color films [1]. Since the early 1990s, aerial photos are commonly acquired with large-format digital cameras and the acquired photos are typically saved in a digital format that can be viewed and edited on computers to assist with modern image processing and analysis [1].

A photogrammetric film scanner, Vexcel VX4000HT, was used to scan (i.e., digitize) the 50 selected historical aerial photos on aerial film rolls. This photogrammetric scanner is recognized worldwide as products of choice for scanning professionals because it can scan large volume film in high definition (up to 1200 dots per inch or dpi) and at high speed (approximately 5 min for a 10 by 10-inch frame). The digitized historical aerial photos were saved in Tagged Image File Format (TIFF) format for subsequent georeferencing operations and accuracy analysis. All 50 selected historical aerial photos on aerial film rolls were scanned at 1016 dpi (25 microns) in TIFF format. It is worth noting that there are no set standards for scanning resolution for aerial film. A resolution of 1016 dpi (25 microns) was used for this project because this is the resolution used by the U.S. Geological Survey (USGS) for their historical aerial photos that are available on EarthExplorer. At this specific resolution, the aerial photos will have a medium or large scale and at the same time maintain a decent level of detail.

Two remote sensing specialists and three student analysts were selected to assist with digitizing, georeferencing, and accuracy assessment. Training workshops for digitizing, georeferencing, and documenting historical aerial photos were developed by the two remote sensing specialists and then provided to the three student analysts. It should be noted that the student analysts’ trainings were not tied solely to this project. That being said, student analysts can use the skills they learned from this project to participate in other geospatial projects or benefit their future careers. They were trained to enter the workforce with an enviable set of skills and a solid body of knowledge, including historical aerial photo digitization, georeferencing, metadata creation, metadata standards, archive/library structure, aerial photo indexing, and professional report and document development, historical aerial photo product management and dissemination, GIS, photogrammetry and remote sensing, web mapping, web application development, and spatial analysis. This is an essential and unique aspect of this project’s sustainability.

3.5. Georeferencing Digitized Historical Aerial Photos

The 50 selected historical aerial photos mentioned in the previous section were georeferenced with the developed online georeferencing tool and two widely used commercial software programs that include ArcGIS ArcMap and ERDAS Imagine. That being said, each of the 50 historical aerial photos was georeferenced three times using the online georeferencing tool, ArcGIS ArcMap, and ERDAS Imagine, respectively. Subsequently, 50 historical aerial photos led to 150 georeferencing practices. Two remote sensing specialists and three student analysts were assigned to conduct georeferencing, with each of them being focused on 10 photos. To ensure a fair comparison, NAIP aerial imagery mosaicked at New Mexico county level was used as the reference image across the three georeferencing platforms because it is very challenging to use Google Maps API in ArcGIS ArcMap and ERDAS Imagine. Noting that this reference image setup was temporary for the online tool with the aim of assisting with georeferencing performance comparison.

Eight GCPs were used to georeference each photo. Noting that this study was not focused on determining the optimal amount of GCPs for georeferencing historical aerial photos, and eight GCPs were used only to complete a second order polynomial transformation. To ensure accurate and fair comparison, the same eight GCPs were used across all three platforms when georeferencing a photo. This process was completed by first georeferencing a photo in ArcGIS ArcMap, and then the identical GCPs were identified and placed in ERDAS Imagine and the online georeferencing tool, respectively.

When selecting GCPs, many principles and best practices were followed to get the best accuracy possible from georeferencing. Firstly, a GCP should be placed on an object on the ground without any obstructions. For example, a GCP should not be placed on an object that is close to a tall object such as buildings or trees that could potentially produce obstructions. Secondly, a GCP should be distinct from its surroundings. In other words, a GCP should be placed on an object that is apparently noticeable and has a high contrast with its surroundings to allow for prompt and accurate identification and placement on the historical aerial photo to be georeferenced and the reference image. Thirdly, GCP should be permanently located on the ground. Considering the historical nature of the aerial photos, when selecting GCPs, priority should be given to objects that will not move over years. Some of the objects that can be used as GCPs include intersections of roadways, large rocks or boulders that stick out of the ground, corners of roadways or sidewalks, and small trees such as shrubs and bushes. Finally yet importantly, GCPs should be evenly distributed across a historical aerial photo. With that being said, GCPs should not be clustered in a specific region or in proximity in a close distance. It is suggested dividing a historical aerial photo into a certain amount of equal size regions (i.e., tiles) and each region should have only one GCP to avoid placing multiple GCPs in a region. Operationally, oftentimes it is impossible to find a valid GCP in one region or even several regions, and in circumstances like this, users are not required to place a GCP in that region or several regions, but instead, they can place two or more GCPs in the same region.

3.6. Georeferecning Acccuracy Assessment

An accuracy assessment of different georeferencing tools was performed once all 50 historical aerial photos had been georeferenced. RMSE was used as the key metric for accuracy assessment. Essentially, it is a measure of how spread out (i.e., standard deviation) the residuals are (i.e., the difference between predicted value and observed value of quantity of interest). In the context of georeferencing, the residual error is the difference between where a GCP ended up as opposed to the location that was specified for it. The total error, or RMSE, is computed by calculating the root mean square (RMS) sum of all residual errors, which can be used to describe how consistent the transformation is between the different GCPs. When the RMSE value is particularly high, users can remove a GCP and add another or a few GCPs to reduce the value. The ultimate goal is to get a low RMSE value and meanwhile maintain the even distribution pattern of the GCPs. A rule of thumb is keeping the resultant RMSE value less than one to three pixels [25]. For instance, the RMSE value should not be greater than three meters if the pixel size is one meter. For this study, each of the 50 historical aerial photo had been managed to get the possibly lowest RMSE value, which is less than one pixel.

Calculating of the RMSE is very straightforward and can be completed in a few steps. The first step is to calculate the residual errors of each GCP, where residual error (E) is defined as the positional difference between a GCP’s actual location and its specified location (i.e., actual location’s coordinates subtract specified location’s coordinates). The second step is to square the errors and sum up all squared residual errors. The third step is to calculate the arithmetic mean of the sum of squared residual errors. The last step is to calculate the square root of the value returned in the fourth step. The aforementioned four steps can be expressed in the equation (Equation (1)) below.

$$RMSE=\sqrt{\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n[{(E_{x, i})}^2+{(E_{y,i})}^2]}$$

(1)

where RMSE denotes root-mean-square-error, $E_{x,i}$ denotes error in x direction (i.e., easting or longitude), $E_{y,i}$ denotes error in y direction (e.g., northing or latitude), and n denotes the amount of observed GCPs, and in this case, it is eight. In the context of this study, $E_{x,i}$ denotes error in easting while $E_{y,i}$ denotes error in northing since the reference image in a projected coordinated system.

Table 1. RMSE of Each Historical Aerial Photo in Three Georeferencing Platforms.

Photo ID	RMSE (m)			Photo ID	RMSE (m)
	Online	ArcMap	Imagine		Online	ArcMap	Imagine
1	1.77	3.70	2.48	26	0.96	1.47	0.61
2	1.69	3.38	3.31	27	2.76	8.29	5.39
3	2.72	3.34	2.75	28	1.53	2.76	1.47
4	7.44	11.48	11.90	29	2.84	8.89	3.85
5	1.72	2.65	1.98	30	1.96	4.34	1.77
6	2.09	5.00	3.13	31	1.84	3.07	2.58
7	1.51	2.45	1.82	32	1.97	3.66	1.15
8	3.07	10.44	7.25	33	4.40	25.33	9.66
9	2.16	5.00	3.49	34	2.66	8.24	3.00
10	2.65	7.96	6.50	35	2.64	7.28	3.06
11	1.82	3.57	2.86	36	1.55	3.17	1.25
12	1.58	3.73	3.78	37	2.97	9.42	4.07
13	2.60	5.13	4.61	38	3.38	12.18	5.22
14	1.59	3.24	2.12	39	2.74	12.73	4.17
15	2.08	4.92	3.71	40	3.11	10.84	5.47
16	1.55	2.30	1.96	41	2.08	2.47	2.41
17	1.31	2.59	1.22	42	0.78	0.37	0.38
18	2.03	4.52	3.11	43	0.89	3.34	0.73
19	2.02	4.76	3.40	44	0.59	2.63	0.86
20	2.03	3.86	2.34	45	0.72	0.78	0.26
21	2.32	5.98	4.75	46	0.79	1.36	0.55
22	3.97	17.29	4.91	47	0.94	1.39	2.16
23	2.81	10.38	7.65	48	3.13	3.62	5.95
24	3.85	15.41	3.99	49	2.04	2.79	1.16
25	3.59	14.98	10.98	50	2.96	2.72	2.75

shows each historical aerial photo’s RMSE obtained from each georeferencing platform. Subsequently, the RMSE values obtained from the developed georeferencing tool were compared with those obtained from ArcGIS ArcMap and ERDAS Imagine.

3.7. Georeferecning Acccuracy Comparison

The sample size was examined to assist with the selection of appropriate statistical tests. In general, statisticians accept that that non-parametric statistical tests should be used if the sample size is less than 30, even if sample values are normally distributed [32,33,34]. For this study, the sample size for each set of RMSE is 50, and therefore, normality test was performed to examine if each set of RMSE is in a normal distribution (i.e., symmetrical, bell-shaped distribution). Three separate tests, including Shapiro-Wilk (SW) test, Anderson-Darling (AD) test, and Kolmogorov-Smirnov (KS) test, were used for normality test to err on the side of caution.

Table 2. Normality Test Results for Each Set of RMSE (* denotes significant at the 0.05 level).

Normality Test	Online	ArcMap	Imagine
Shapiro-Wilk	<0.0001 *	<0.0001 *	<0.0001 *
Anderson-Darling	0.0071 *	<0.0001 *	0.0004 *
Kolmogorov-Smirnov	0.0435 *	<0.0001 *	0.0154 *

shows the normality test result for each set of RMSE at a significance level of 0.05.

For each test, the null hypothesis is that the distribution of a set of RMSE is normal with unspecified mean and standard deviation at a significance level of 0.05. As shown in Table 2, the p-value for each test is less than 0.05, and therefore, the null hypothesis should be rejected, indicating that the distribution of the RMSE is not normal. Therefore, non-parametric statistical tests, which do not assume anything about the underlying distribution such as normal distribution, were used for this study.

The assessment of the georeferencing accuracy the three platform was trifold. First, a box plot was created to graphically exhibit the locality, spread, and skewness of the each set of RMSE, highlighting the mean and outliers. Second, a matched group (all three platforms) and pairwise comparison (i.e., online tool vs. ArcMap, online tool vs. Imagine, and ArcMap vs. Imagine) were performed to understand which platform consistently provides the highest accuracy or lowest RSME. Third, formal statistical test was performed and descriptive statistics were derived to examine statistically which georeferencing platform provides the highest accuracy or lowest RMSE. Specifically, the matched group and pairwise comparison were refined with formal statistical analyses of the distribution of RMSE among all three platforms and between each pair of platforms. The non-parametric Friedman test was used to examine if there is a difference in central location (median) among the three sets of RMSE. The non-parametric Wilcoxon Signed Rank test was used to compare the distributions of paired georeferencing platforms. In addition, the distribution of each RMSE set was summarized by their means, medians, and standard deviations.

4. Results and Discussion

As a first analysis step, a box plot was created to assist with visual analysis and comparison among the three georeferencing platforms. Figure 9 shows the box plot and associated median, mean, max, min, the first quartile, the third quartile, and the outlier values. The x-axis indicates the three georeferencing platforms while the y-axis indicates the RMSE. The plot area indicates the distribution of the RMSE of each platform. As shown in Figure 9, both the median value and mean value of the RMSE set of the online tool were less than ArcGIS ArcMap’s and ERDAS Imagine’s values, indicating that the online tool can be used to produce georeferenced images that have higher accuracy. In other words, this observation reveals that the online tool outperforms ArcGIS ArcMap and ERDAS Imagine in terms of georeferencing. In addition, both the median value and mean value of the RMSE set of ERDAS Imagine were less than ArcGIS ArcMap’s values, implying that in general the ERDAS Imagine has better georeferencing performance or higher accuracy. Surprisingly, the geospatial industry widely adopted software ArcGIS ArcMap has the poorest performance in terms of georeferencing.

Further examination revealed that the online georeferencing tool yielded the least amount of outliers (one) with the lowest values (7.44), when compared to ArcGIS ArcMap and ERDAS Imagine. When compared to ArcGIS ArcMap, ERDAS Imagine yielded more outliers (three) but their values were much smaller (11.09, 10.98, 9.66 vs. 17.29, 25.33) than the outliers yielded by ArcGIS ArcMap. Essentially, this indicates that the online georeferencing tool outperforms ArcGIS ArcMap and ERDAS Imagine in terms of creating large positional errors due to a variety of reasons such as misplacement of GCPs, incorrect selection of GCPs, and inappropriate distribution of GCPs. Likewise, ERDAS Imagine outperforms ArcGIS in terms of creating large positional errors.

Continuing with visual analysis, formal statistical tests were performed to investigate: (1) if the visual observations were statistically significant, and (2) statistically which georeferencing platform provides the highest accuracy or lowest RMSE. As mentioned in the previous section, two non-parametric statistical tests, including the Friedman test and the Wilcoxon Signed Rank test, were used to compare the three georeferencing platform to investigate which one can provide the highest accuracy.

The Friedman test, which is a non-parametric alternative to the repeated measures ANOVA test, was used to examine if there are significant differences in median values among three or more matched groups, and in this case, the three sets of RMSE that were derived from the three georeferencing platforms. For this test, the null hypothesis is that the medians of the three sets of RMSE are all equal at a significance level of 5%. Test results revealed that the p-value was much less than 0.05, and therefore, the null hypothesis should be rejected, indicating that the medians of the three sets of RMSE are not equal at a significance level of 0.05. Hence, significant differences in median values were found among the three platforms. That said, there are performance differences among these the three georeferencing platforms. This result corresponds to the above visual observations.

The Wilcoxon Signed Rank test, which is a robust non-parametric alternative to parametric t-test, was used to examine there are significant differences in median values in a paired group, and in this case, any two sets of RMSE that were derived from the three georeferencing platforms form a paired group. Therefore, three Wilcoxon Signed Rank tests were performed since there are three paired groups.

Table 3. The Wilcoxon Signed Rank Test Null Hypothesis and Result (* denotes significant at the 0.05 level).

Paired Group	Null Hypothesis	p-Value
Online vs. ArcMap	The median difference between the two sets of RMSE (Online, ArcMap) is zero at a significance level of 0.05	<0.0001 *
Online vs. Imagine	The median difference between the two sets of RMSE (Online, Imagine) is zero at a significance level of 0.05	<0.0001 *
ArcMap vs. Imagine	The median difference between the two sets of RMSE (ArcMap, Imagine) is zero at a significance level of 0.05	<0.0001 *

lists the three paired groups and each one’s corresponding null hypothesis and test result.

As shown in Table 3, all p-values were less than 0.05, and therefore, the null hypothesis for each test should be rejected, indicating that the median difference between any two sets of RMSE is not zero at a significance level of 0.05. Hence, significant differences in median values were found between any two sets of RMSE. That being said, there are performance differences between any two of the three georeferencing platforms. Since the alternative hypothesis is not explicitly characterized in non-parametric tests, only further inspection of the distributions’ descriptive statistics can provide valid information to investigate which georeferencing platform has the best performance for each paired group. Therefore, mean, median, and standard deviation of the RMSE for all georeferencing platforms were computed, summarized, and analyzed in

Table 4. The Descriptive Statistics for Each Georeferencing Platform’s RMSE.

Georeferencing Platform	RMSE Descriptive Statistics
	Mean	Median	Standard Deviation
Online	2.28	2.06	1.15
ArcMap	6.02	3.80	4.94
Imagine	3.52	3.30	2.58

.

As shown in Table 4, the online tool always had lower values for mean, median, and standard deviation, followed by ERDAS Imagine, while ArcGIS ArcMap always had the highest values for mean median, and standard deviation. This result, coupled with the result that there are performance differences between any two of the three georeferencing platforms, verifies that the online tool has the best performance in terms of georeferencing accuracy, while ArcGIS ArcMap has the poorest performance, and ERDAS Imagine falls in between. To continue the pairwise comparison, a summary table was also created with the aim to examine which georeferencing platform has a better performance for each of the 50 historical aerial photo. The results were summarized in

Table 5. Pairwise Comparison between Each Georeferencing Platform.

Paired Group	Comparison Results
	Online Better	ArcMap Better	Online Better	Imagine Better	ArcMap Better	Imagine Better
Online vs. ArcMap	48 (96%)	2 (4%)
Online vs. Imagine			38 (76%)	12 (24%)
ArcMap vs. Imagine					6 (12%)	44 (88%)

.

Table 5 revealed that when compared with ArcGIS ArcMap, the online tool performed better (higher accuracy) for 48 out of the 50 historical aerial photos, which represents a percentage of 98%. This proves that when compared with ArcGIS ArcMap, the online tool consistently outperforms ArcGIS ArcMap. For two of the 50 historical aerial photos, ArcGIS ArcMap performed better, and further inspection revealed that the two photos did not have any noticeable characteristics. However, these two photos were georeferenced by the same student analyst, and the poorer performance of the online tool could be caused by the student analyst’s selection of GCPs. Additionally, when compared with ERDAS Imagine, the online tool performed better (higher accuracy) for 38 out of the 50 historical aerial photos, which represents a percentage of 76%. For 12 of the 50 historical aerial photos, ERDAS Imagine performed better, and further examination revealed that among the 12 photos, six of them were georeferenced by a specific student analyst, while the reminding six photos were referenced by one remote sensing specialist (two photos) and two student analysts (one and three photos, respectively). This observation suggests that the better performance of ERDAS Imagine may be attributed to the varied interpretation and selection of GCPs by different users. Another observation is that among the 12 photos, eight of them were taken over relatively mountainous terrain, which may make it more challenging for users to select the most proper GCPs. However, one unique feature of ERDAS Imagine is that it allows for deeper zoom in on both the raw photo and reference image when compared with the online tool and ArcMap ArcGIS. Operationally, this unique feature may be able to provide more practical assistance with GCP interpretation and selection for historical aerial photos that have relatively mountainous terrain.

The georeferenced historical aerial photos that were produced by the three georeferencing platforms were also visually inspected and patterns were summarized (Figure 10). One of the most noticeable patterns is that both ArcGIS ArcMap and ERDAS Imagine use a different warping or bending strategy than the online tool. Additionally, there is noticeable difference between ArcGIS ArcMap and ERDAS Imagine in terms of warping or bending. The online tool’s better performance or higher accuracy could be attributed to the distinctive warping strategy provided by GDAL. Another observed pattern is the three platforms use different image stretching methods, with the ERDAS Imagine being the brightest and the online tool being the darkest. In general, the online tool can provide the highest accuracy or lowest RMSE values for georeferencing, but at the cost of visual fit.

5. Future Research Directions

Future research using the developed online tool could include any area that conduct georeferencing that would otherwise be completed with commercial or free and open source desktop software programs. Historically, georeferencing was primarily performed with these desktop software programs, and therefore, many of the contemporary concepts or theories such as volunteered geographic information (VGI) or crowdsourcing related research has not been able to be done in the context of historical aerial photo collections. With the help of the developed online georeferencing tool, geospatial researchers can design a large-scale study to examine the effectiveness of crowdsourcing in terms of historical aerial photos. Additionally, educational researchers can leverage the developed online georeferencing tool to study if geospatial web applications can assist with effective Science, Technology, Engineering, and Math (STEM) education practices.

Another research focus is developing mobile applications that can be used to digitize aerial photos. Such mobile applications can use the camera of a smart phone or tablet to take pictures of historical aerial photos to digitize it and upload it to the developed web application. This tool can be integrated with the online georeferencing tool to create a seamless process to complete digitizing, georeferencing, documentation, and web-publication. Additionally, research should be focused on leveraging geospatial artificial intelligence (GEOAI) techniques to automate the georeferencing process. Users only need to upload a raw photo and provide a reference image that is in proximity to the raw photo, while a GEOAI-based web application will be able to detect and identify a proper set of GCPs, select and execute an appropriate transformation algorithm to complete the georeferencing process, and produce the georeferenced photo at a designated directory.

6. Conclusions

Digitized and georeferenced historical aerial photos can provide valuable and irreplaceable information for many applications and fields. To address the limitations of desktop georeferencing software, this study developed a custom-built online georeferencing tool to enable georeferencing digitized historical aerial photos in a web environment. The performance of the custom-built online georeferencing tool was evaluated and research results revealed that the custom-built online georeferencing tool outperformed both ArcGIS ArcMap and ERDAS Imagine in terms of georeferencing accuracy (i.e., RMSE). In addition, ArcGIS ArcMap had the poorest performance and ERDAS Imagine fell in between. Reviewing of the georeferenced images revealed that the online tool provided the highest accuracy or lowest RMSE values for georeferencing, but at the cost of visual fit. Due to the web-based and no-cost nature of the custom-built tool, it holds the potential to maximize its accessibility to assist with STEM education projects and crowdsourcing projects that focus on georeferencing aerial photos. These findings lay the foundation for future research into GEOAI-based georeferencing, which involves developing a web application that can be used to detect and identify a proper set of GCPS, and then select and execute an appropriate transformation algorithm to complete the georeferencing process with limited human interference such as uploading the image to be georeferenced.