Build good habits now
A general reminder that GIS Software does crash, especially if we accidently open files that are too large. Remember to save often; and when (if?) the software crashes, this is not entirely unexpected and the most successful path forward will be to stay as calm as possible. Typically it is while working with particularly large or complex data sets that the software is most likely to crash. This can lead to a false sense of security as the software is stable when we have a low time investment, and will tend to crash after we have an hour of unsaved edits. Save often!
Adding Data to Our Maps
In today’s lab, we are going to explore the various kinds of data that we can add to our maps. Broadly speaking, this data comes in two main types: Vectors and Rasters. Vectors come in Points (waypoints), Lines (roads or tracks from a GPS), and Polygons (outlines such as lakes or buildings). Rasters are imagery: this could be air photos or other data that has been rasterized. For example, the Lidar we used in the first lab started as point data and was rasterized into the digital elevation model we used.
Start a Project
As before, create a new project in ArcGIS Pro called Lab3 inside of your GEOG205 folder on your K: drive.
Use the catalog pane and add L:\GEOG205\lab03 as a folder connection.
Vector Data
Vector Data will be the primary source of data we will be using in this class and we will be looking at various ways of adding it. Our primary motivations for using vectors are that we generally get smaller file sizes than rasters, the ability to store metadata about individual features, and a great deal of control over how to symbolize the data.
Shapefiles
Shapefiles are an ageing file format but still can be thought of as industry standard, as nearly all GIS software will open a shapefile.
Take a look in L:\GEOG205\Lab03\shapefiles\unbc inside the catalogue pane of ArcGIS Pro. You will see a variety of files listed. However, what will be perhaps more interesting is opening this same folder in the Windows file explorer: you will notice that there are in fact many more files here, as a single shapefile is stored in multiple pieces.
Initially, you will probably see multiple files with the same names. In the View tab at the top of the file explorer ribbon, make sure the the box is ticked for ‘File name extensions’ in the ‘Show/hide’ section. Now you’ll be able to see the file extensions for all the files in the folder:
- .shp – File containing geometry
- .shx – Positional index of feature geometry
- .dbf – Contains attribute data for features
- .prj – File containing projection information (optional, but highly recommended)
- .sbn – Contains spatial indexes (optional file, but dramatically increases draw speed in large layers)
Inside of ArcGIS these are all presented as a single file, however, when interacting with shapefiles outside of ArcGIS, care needs to be taken to keep these files together, and if one is renamed, they must all be renamed with the exact same name.
Add the features.shp, roads.shp, and lakes.shp to your ArcGIS project from the Catalog pane.
“Features” is a very generic name, so to help keep our table of contents clean, double click the Features layer in the drawing order panel and rename it to “UNBC Features”
The data we just added comes in three types of geometry: points, lines, and polygons.
Each layer can only represent a single type of geometry. If we have a mixed data set, such as rivers where large rivers are represented by polygons, and creeks as lines; we would either need two layers or convert the geometry to be the same type for all features. An example of this conversion would be to buffer the small river lines to make them polygons, or to covert large river polygons to lines representing the banks (we saw an example of large rivers defined by lines in the rivers_093G layer that we used in Lab 02).
GPX
GPX or GPS Exchange format is a file type you may come across while working with data collected with handheld GPS Units. In comparison to Shapefiles, the GPX is a single file, and can contain all three geometry types at the same time (you will still only be able to load one geometry type per layer).
Despite this, ArcGIS Pro will not directly open these files in many cases (it will only load point features). For this reason, we will Open QGIS.
Once you have QGIS open, in the windows file explorer go to the folder L:\GEOG205\Lab03\gps_data and then drag and drop unbc_biodome.gpx onto QGIS.
The dialogue presented shows the layers inside of the GPX files as well as the number of features in each layer in brackets at the end of each description. In this case, there is a track (recording of where the GPS was) represented as both a line (with 1 feature) and points (with 124 features), in addition to 3 waypoints. Click Select All, then Add Layers.
In QGIS you will see the lower-left corner has a layers pane that works much the same way as Drawing Order in ArcGIS Pro. You can turn the layers on and off with the tick boxes to change what you see on the map.
Right click on the waypoints layer >>> Export >>> Save Feature As
Save as an ESRI Shapefile, and using the … button to the right of the File name browse to your lab3 folder and save as unbc_biodome_waypoints.shp
Do the same for the track points and track line layers and then add all three layers into ArcGIS Pro.
KML
KML is Keyhole Markup Language (KMZ is a compressed zip version of KML) and is the primary file type used by Google Earth, due to the ease of access of Google Earth KML / KMZ files are generally common in areas where data is produced without the need for a full GIS system. KML has an extra advantage over the other files we have looked at so far, in that they also contain a very basic amount of styling information in the form of line colour, and in the case of polygons, fill colour.
Add L:\GEOG205\Lab03\kml\Northern Sports Centre.kmz to your project and notice that the polygon automatically appears in UNBC’s Green.
Tabular Data
It is often common to get data in the form of spreadsheets, and one place to get such data is GeoNames. https://www.geonames.org/
On this website under Downloads, click on Free Postal Code Data, then download CA_full.csv.zip. Find the file in your downloads folder, and extract the zip file into your lab3 folder.
Open Excel, select Open, then click Browse. Change the file type filter from ‘All Excel Files’ to ‘All Files.’
Navigate to and open the CA_full.txt file you just extracted. You will be prompted with a dialogue to help Excel understand what type of data is contained within the file.
On the first page, make sure that the Original Data Type is set to Delimited, then proceed to Next.
On this page, we need to make sure the data preview looks like a spreadsheet. In this case, our Delimiter is a tab, and because there are some blank cells, we need to make sure ‘Treat consecutive delimiters as one’ is disabled so our data is not shifted after an empty column.
At this point you can click finish.
To help ArcGIS understand this data, we will click Save As, and save it as a CSV (Comma Separated Values) file. Essentially, we have just replaced Tabs with Commas in the file.
Now open this file in ArcGIS Pro. You will at this point notice that it does not show up on the map, but rather at the bottom of the Drawing order as a “Standalone Table”. ArcGIS Pro knows it is a table, but not how to interpret the data. Right-click on CA_Full.csv in the drawing order and choose Open. The table will appear below the map. Scroll to the side and make note of which Fields contain the Longitude and Latitude – we will need this information in the next step.
To make it show up on the map, we need to add XY Point Data.
We will then tell ArcGIS which Fields contain the positional data.
As an alternative, you could add the .txt file to QGIS directly and then export it in the same way we converted the GPX File before. There is always more than one way to solve a problem!
Before Proceeding, right click on the Postal Code layers and remove them from your project.
This CSV layer is relatively large and needs to be removed before proceeding.
Databases
The final form of vector input we will explore is spatial databases. These range from file-based databases (Geodatabases, Spatialite) to Enterprise Database Management Systems DBMS (Such as Enterprise Geodatabase, PostgreSQL, MS-SQL with spatial extensions).
You have already been interacting with an Enterprise Geodatabase in this course while using the ArcGIS Portal. The ArcGIS Portal is storing all of its data in a PostgreSQL database; these systems are the highest performance option for storing data and provide the ability to work with large, rapidly changing data. Outside of the portal we also operate PostgreSQL servers on campus that store and serve TRIM, OpenStreetMap (entire globe), and BC Vegetation Resource Index, all data sources that exceed the file limitations of Shapefiles. That said, you are unlikely to come across such systems outside of large organizations or smaller business that focus specifically on geospatial data.
File-based geodatabases are the default method in which ArcGIS Pro stores data, in fact, if you look in the Catalog pane you will find that there is already a lab3.gdb file in your lab3 folder, this is the geodatabase for your project, anytime you export data it will be placed into this file. Inside of ArcGIS, a geodatabase looks a lot like a folder, but to windows explorer, it is a single file. These are sometimes used to send someone multiple layers at once.
Rasters
The other broad category of data that you will run into is raster, possibly more familiar when it’s referred to as images. Common examples of this would be air photos, satellite imagery, scans of paper maps, and digital elevation models.
We have already added raster layers to our maps in this class when we added the UNBC Campus orthoimage, and the paper NTS map sheet. But now we are going to look at a scenario you may run into if an image either doesn’t contain spatial information, or if this information is inaccurate.
Before we begin, set the base map to Imagery so we have something to reference against:
In the File Explorer, copy L:\GEOG205\lab3\raster\campus20cm_2010.jpg to your lab 3 folder, and open the copy in ArcGIS Pro. Allow Arc to calculate statistics if it prompts you.
Make sure your Map is set to ‘WGS 1984 UTM Zone 10N’ by right-clicking on Map in the Contents pane, then Properties > Coordinate Systems, and verify the Current XY setting. If it is not correct you can find the projection under Projected Coordinate System >>> UTM >>> WGS 1984 >>> Northern Hemisphere >>> WGS 1984 UTM Zone 10N.
The data we added should have appeared to the northwest of the campus somewhat skewed (or it may appear in the middle of the Pacific Ocean). To make this show up properly, this image must be ‘georeferenced’.
Select the image in the drawing order. Then go to imagery and georeference
This is a tool that we can use to show ArcGIS Pro where the imagery belongs. You may also need to do this process if the data is perhaps just poorly aligned.
This is going to be done in a 2 step process: 1st prepare, 2nd adjust (as a note if the image imported was just poorly referenced the prepare step could be skipped).
A great place to start is to zoom in on the campus, and press ‘fit to display’, this will cause the imported image to at least be in the general area of UNBC. You can then use the Move and Rotate tools to drag the image to the approximate location. Tip: Make sure the roads layer is still turned on and above the image in the drawing order so you can use it as a guide.
Next, we are going to Add Control Points, control points are created in pairs, first on the image we want to adjust, and second on the target or what we want to match against.
Click on Add Control Points, your cursor should change to “From point (source)”
Zoom in on UNBC and click as close as possible to the center of the campus compass between the buildings.
At this point your cursor should change to “To point (target)”
Hide the image you are trying to adjust in the Drawing order and click on the same point on the basemap.
When you turn back on the image you are adjusting you will notice it will snap into place. However the image may still have the wrong rotation or scale, to fix this we will add more points. You will have noticed that your cursor is once again at “From point (source)”
Pick 4 more control points to make. Some guidance in order of importance
- Control points must be in the same place between images
- Control points should be spread apart and distributed across the image
- Control points should be at ground level whenever possible, avoid trees, poles, and buildings as these can change with the camera perspective. If you must use these features pick a point at their base.
You should end up with something similar to this:
Notice how the road overlay is on top of the roads and relatively lined up.
When you are happy with your results select Save as New in the Georeference tab, then save it in your Lab 03 folder using the TIFF file format.
What will be exported is a GeoTiff, which is a Tiff file with coordinates in the metadata allowing this image to be opened in GIS software without the need to be georeferenced again.
Some potential sources of raster data might be:
PG Map: https://pgmap.princegeorge.ca/Html5Viewer/index.html?viewer=PGMap
If you are living outside of Prince George, your municipality may provide a similar service.
For Satellite images: https://earthexplorer.usgs.gov/
Editing Data
Something you will very quickly discover, if you have not already, is that spatial data often contains errors that need to be fixed before they are suitable for mapping. This can come in several forms such as missing data that needs to be added, anomalous data to be removed, or data that just needs a little bit of an adjustment to its precise location.
Never edit original data – it’s always good to be able to go back if you need to. Let’s make a copy of the roads layer: Right click >>> data >>> export features.
The only setting you should need to change is to rename your output file: Click in ‘Output Feature Class’ and notice where it is saving your file. At the end of the file path, after ‘[YourLabName].gdb\’, add your new file name – call it unbc_roads. By default, it will save into your project’s (Lab3) geodatabase.
For this next task, select your new roads layer and then the edit tab. Let’s start with a quick chat about the buttons available in the editor.
Selection tools:
Before we can edit features, we need to select what we want to edit. There is a drop-down menu under Select for picking different selection tools when a rectangle selection is just not what you are looking for.
Attributes can be used to select features by values in the data table. For example, maybe you want to select all the creeks in a layer with watercourse data. This is a feature that is better covered in GEOG 204 and GEOG 300.
Finally, there is a Clear button. It can be useful to always click this before selecting new features to make sure you are not editing features you have already finished with.
Clipboard:
The tools in the clipboard section work pretty much as you would expect, it is somewhat unusual to paste within the same layer, however, one way to move features to a different layer is to cut, select the other layer in the drawing order, then paste.
Manage Edits:
Here are two of the most important buttons: when we are finished editing, Save writes all the changes we made back to the original file. Discard is useful if you make a mistake that might have been too far back to undo actions all the way until you find the error, or you want to completely restart your edits. Discard will bring the layer back to the state it was in when you last clicked Save.
Features:
This is the last section of the top ribbon we are interested in. Both Create and Modify will open new panels on the right side of the screen, over top of the catalogue. Delete will simply remove whatever features are selected.
Create:
When the Create Features pane opens, it will show you a list of editable layers. Select the layer you wish to add features to, and you will be presented with a set of tools appropriate to the type of geometry contained in that layer.
Take a few minutes to try out the various tools. Do not Apply/Save your edits, we will discard them before moving on. The idea here is to get a feel for how the tools work.
Snapping:
Snapping is useful when we want to make edits without gaps. Snapping will assign our cursor coordinates to exactly match nearby features. Try turning it on, and hover your cursor over a road feature. Does the cursor change to show what you are hovering on?
What happens if you hover over the middle of a line as opposed to the end?
Adding a road to the campus:
Make sure snapping is turned on, and add a road to the roads layer leading to the power plant on campus using the line tool, it should look like below. Double-click when you are done drawing your line. Ask your TA any questions you run into during this process.
Modify:
Modifying features is possibly the most finicky process in the class. It’s not hard, but mistakes are common and there is no shame in the discard button and starting over!
Let’s clean up the road in the bus loop to make it better match the background.
Click modify
Then in the Modify Features panel that appeared on the right, find the Edit Vertices tool under Reshape.
Then click on the road in the bus loop. Once selected, a new toolbar will appear at the bottom of the map.
The first tool from left to right is the lasso selection tool, you can use the drop-down to pick a different selection tool if you prefer.
Draw a selection over several vertices, then hover over one of the selected vertices until your cursor changes to “roads: Vertex” at this point you can click to drag the nodes.
The next tool is the Add Vertex tool, with this tool selected you can click on a line segment to add a vertex at that point and then click and drag to make corrections.
Following that is the Remove Vertex tool, with this tool selected simply click on a vertex to remove it.
Finally in this section is the Continue Feature tool, this is much like the add vertex tool except instead of placing a vertex between existing vertices, clicking on the map will create a vertex and connect it to the most recently created vertex in the geometry.
Spend some time cleaning up the road and getting used to the tools.
Don’t forget if you make a mistake you can undo it with [Ctrl] + [z], or the back button in the upper left of ArcGIS Pro. Or, make use of the Cancel button at the end of the extra toolbar to undo all your recent edits.
Lab 3 Assignment
For this week’s lab, you will be making a map. Your map will consist of several layers that you will need to import into your project and edit. At a minimum, your map should also contain:
- Title
- Scale bar (with logical segments)
- Your name and the date
- Vector layers should be identifiable (ie. different colours and a legend) – if it’s not perfect, don’t worry, we are doing symbology next week!
Your map will cover the trails near Shane Lake with an extent similar to the image below.
1. Add layers to the map
In the folder L:\GEOG205\Lab03\assignment_data you will find
- unbc_pNeo_2021_RGB.tif (add this to your map from here)
- trails shapefile layer (copy this to your K: drive from the L: drive as we will be editing it, and we don’t edit original data!) – be sure to copy all the trails files, not just trails.shp. [You could alternatively add this vector layer from L: to your project, and use the dropdown (right-click on it in table of contents) and choose data-> export features and save this new shapefile in your lab folder on the k: drive. Then, remove the original shapefile from your project and add your copy ]
- bird_nest.csv
2. Add the bird nest data as XY Point Data
Scroll back in the lab instructions for a reminder how to do this! Remember that clicking on the coloured point, line, or rectangle beneath the layer you want to edit will open the Symbology pane that will allow you to change how the layer is displayed, and also remember that zooming in and out on your map will force the map to refresh.
3. Edit the trails layer
See image above for trails to be removed – four (marked with an X), and trails to be added – four (shown in red)
4. Find Additional Data
Part of working in GIS is locating the data sets that you need. Use the City of Prince George website and find the layer “Water Structure” (This includes the water tower in Forests for the World) and add it to your map.
5. Create a new layout
Create a new layout in ArcGIS Pro, and make a Letter Sized Portrait* map.
- In fact the area needed is better suited for landscape format. If you haven’t yet created the layout, feel free to use Landscape as it will be easier to fit the map area. You are being graded on the editing part of the lab, not the layout (which we’ll go through in Monday’s lecture), so no worries either way.
6. Add additional information to the map
Title, scale bar, your name, date.
7. Save your map as a PDF
Check out the Lab 01 instructions if you need a reminder of how to do this. Submit the PDF file of your assignment to your TA via Moodle by the beginning of your next lab.