Facilities Information and Technology Services

Workflow- Processing Imagery and Tile Caching

Topic:
Enterprise GIS
Tags:
Drone

Overview

This is a guide to taking raw drone data through several steps that will result in an image service using the data.

This is an advanced version of the KB "Generic Postflight Processing in SiteScan".

This should only be necessary on very large flights and the prior KB would suffice for many small flights not requiring a dedicated service or tile cache. 

Most commonly done with the Wingtra and for the Leaf On/ Off Imagery collection of the Columbus Campus.

Step 1. Geotagging

Start by converting images into geographically aware images.

  1. Add all drone imagery to the file directory in "Drone Flights", K:\AP\Applications\GIS\Projects\Drone Flights
  2. Ensure your imagery conforms to the standard file directory format. 
    • This can be copied from K:\AP\Applications\GIS\Projects\Drone Flights\!!Flight Documentation\"_Pix4D Folder Structure - Copy This"
    • Paste this structure in your drone imagery folder and move raw data into the folder "images"
    • Paste other data, ground control files etc to the folders needed AARON WHAT ARE THESE?
  3. Open wingtra hub on the beast, and geotag

Step 2. Process Imagery

It is time to merge 100s of images and create a tiff file (or multiple).

  1. Open Sitescan and create a new project 
  2. Upload a images on a per-flight basis AARON HOW DID YOU GROUP THESE MERGES?
  3. Process the images
    • add settings used here
  4. review the images for issues and reprocess major holes, deformities, etc
  5. download the tiffs to a working directory. 

Step 3. Clean Tiffs

It is time to combine the data and make it nicer to look at.

  1. Add the handful of merged tifs processed to ArcGIS Pro
  2. Build Pyramids for each tif. This may automatically be done but this is with the incorrect settings most of the time.
    1. Ensure the correct values by rebuilding with:
      1. Resampling Technique set to: Bilinear. This is to smooth the "static" look found in nearest neighbor sampling
      2. Compression type: default. Not sure of the other types but "no compression" adds a black border to the data. 
      3. Environments- Parallel Processing on a strong computer speeds up processing time. 
    2. Doing this may cause a large black boundary on the tif. Try removing this using the symbology pane "mask" tab. 
  3. Layer order decision making. This is up to the user but the best imagery should be on top of the adjacent imageries to be the image taken by caching.
  4. Drag around the unaligned tifs if you used multiple drones for collection with varying x,y accuracy.   
  5. Mask the overlap and clean up borders.
    1. Simple masking can be done using a few feature layers and no geoprocessing tools. 
    2. Ensure masking is an option on the Raster Layer tab within Pro. Select the feature layer you make to omit the overlapped raster data from view.
  6. You do not need to merge the tifs into one to make a cache. You will get a "what you see is what you get". Ensure what you see looks good or else the cache will not. 

Step 4. Create a Tile Cache

A static copy of the data optimized for draw speed needs made. 

  1. Use the Manage Tile Cache GP tool to make a cache. 
    1. recreate all tiles. 
    2. run a medium-size layer alone to verify the output looks correct first.
    3. It tries to make you select a single tif. You can go to your project and select the whole map. Avoiding the need for a merge of the layers.  
  2. Run all layers
  3. Add the cache to a map and QC it at many locations and zoom levels. Slowly so the project has time to redraw according to the cache.
  4. Move the image cache to the GIS file server following the folder conventions 
  5. Remember the location of this cache folder for later. It will be referenced.

Step 5. Create Image Service

Define a service and alter it. 

  1. Make a service using "share web layer". In configuration you should be able to establish a connection to an existing tile cache. 
    1. If not, rename the folder so the cache is not accessible ie "cachename_temp"
    2. publish the service with a dynamic reference to the data. 
    3. go to server manager and change the reference type to from cache and let it build a folder directory. 
    4. paste the data in from the temp folder.
  2. Activate the service
  3. paste the correct tiles into the imagery folder directory. 

Step 6. Clean Up

  1. The files created over the course of this need reviewed and consolidated. 
    1. This process can make hundred of Gigabytes of data if you are not careful. 
  2. Raw processed imagery can be discarded if it has not been altered. It can be retrieved later from SiteScan if needed. 
  3. Keep the copy of the imagery that you performed the build pyramids and masking on for the current year.
    1. Go back a year and delete the previous one, this is only kept in case an issue arises and the cache needs rebuilt to tweak something. 

Done!