Making use of text data, tables and digitizing

Learning objectives:

  • Importing and displaying coordinates from a csv (text) file
  • Joining tabular data to spatial data
  • Saving data to different formats
  • Creating lines from points and polygons from lines
  • Creating new shapefiles
  • Digitizing data

Taking tabular data and bringing it into QGIS

Data Management

Make a folder for today’s lab to save all your data into (i.e. lab2 in your geog204 folder). Once it is created, copy the following data from the L:\GEOG204\labs\lab2 folder:

postal.txt
theftsfrom.csv

Cleaning the data

Right click on the postal code text file and open with Notepad on Windows. Take a look at the data and see if you can figure out what is going on.

  • Clean off the metadata, set up the column names to be postal lat and lon and save the file in your folder.
  • What are the commas used for?

Open the file theftsfrom.csv in both in a text editor and in a spreadsheet program (e.g. Excel or LibreOffice Calc).

  • Do you understand how the data is put together? Do the commas from the postal code dataset make more sense?
  • What type of data is being recorded?

Bringing text data with and without coordinates into QGIS

Open a new blank project in QGIS. We are familiar with shapefiles and we should know how many files make up a shapefile, but the files yo have looked at today are simple text data. Data like this can be loaded in such a way to create point features from plain text.

Open the postal code data by:

  • Layer –> Add Layer –> Add Deliminted Text Layer –> choose your postal code file you cleaned above
  • Set up the import panel to make use of the x, y coordinate in the file
  • What are the x and y coordinates? The image below should help you out.

What does this data show us?

You can follow a similar process to add the thefts data, but since there are no coordinates to reference each features’ location in space, it is just a table – select the option that tells QGIS to not consider the file a spatial layer (no geometry).

Base map – and plugins

Plugins

We have looked at QGIS a fair bit it the last two labs, but we have yet to look at one of the great features of the software – Plugins

QGIS allows people to write their own plugins, which adds functionality to the software. Use the menu along the top of QGIS window to Manage and Install Plugins.

  • Go to the settings tab and turn on all the check boxes so you can see experimental plugins also
  • Go back to the All tab and search for the QuickMapServices plugin and then Install Plugin

Plugins sometimes get added to odd places, but you can find this one on Web Menu –> QuickMapServices

Once you are there add the contribution pack to the plugin by
settings –> more services –> get contributed pack –> save

We have two data sets in our TOC (table of contents). We can add a background map to get some reference using teh QuicMap plugin. If you go to the plugin under web, choose a background that you like (i.e. OSM–> OSM standard or ESRI–> ESRI Topo)

Bringing the datasets together – they are much more useful this way

We want to be able to see where each vehicle from the theftsfrom was stolen, but that file has no geometry with it. What we do have is points for each postal code. Is there something in common that will allow us to join them together?

Open the attribute table for both layers so you are familiar with the attributes that are listed.

Joining Tables in QGIS

Right click on the thefts layer–> properties –> joins tab

  • Add a join (plus button)
  • Select the common fields
  • Check “common field name prefix” –> set it to empty
  • Click Apply
  • Click OK

Open the attribute table and see the difference – there should be lat/long values listed

Now that the table is joined – save the thefts from layers as a CSV type and bring it back in as a spatial layer.

Right click thefts –> Export –> Save Features As

  • Format Comma Separated Value
  • Choose a name and location

Even though it adds it in automatically after you save it, it doesn’t recognize the geometry unless you add it back in. Since it now has coordinates in the attribute table, you can add it as a CSV with coordinates.

Now save both the postal layer and the thefts point layers as shapefiles in your folder.

Reviewing your data and your thought process

It is always important to take a critical look at your datasets and outputs. Look at the distribution of your points, does it make sense when you compare the postal codes to the thefts? Does it matter that there are more thefts than postal codes?

Question 1: Why isn’t the postal code a unique identifier in this thefts shapefile? Does each postal code have a theft? Is the theft point the ACTUAL location where the theft happened, what is your reasoning behind this? What about thefts that happened in the same postal code – how do they get displayed?

A quick way to add Unique Identifiers in QGIS

Lists of attributes like this should have a unique identifier. It is a good practice to create unique IDs. Often we are working in the geometries of a layer or features, but we also spend much of our time altering the attributes of each layer. In this case we are cleaning up the data through its attributes by adding a unique ID

Open the attribute table –> open field calculator –> new field (call it uid perhaps) –> enter the “@row_number” function in the expression panel –> Click OK

Creating lines from text files

Line generate files

We just created a point feature type layer. This file did not have a unique ID for each location associated with each row’s x,y coordinates, but rather postal codes (that are not necessarily unique) so we assigned it a unique ID. The data we started with were stored in a text format using deliminator such as commas and is often called a generate file –we were generating a point layer (one feature per row of text). This concept can be extended to creating lines and polygons from text using the Simple Features Model.

It is possible to create lines from personally digitizing (connecting points) points, but it is also possible to generate a line by connecting points (or vertices).

Generate file for lines:

12
0.5121700E+06 0.5971884E+07
0.5121580E+06 0.5971892E+07
0.5121380E+06 0.5971892E+07
0.5121310E+06 0.5971972E+07
0.5121270E+06 0.5972053E+07
0.5121840E+06 0.5972065E+07
0.5122610E+06 0.5972076E+07
0.5122700E+06 0.5972009E+07
0.5122740E+06 0.5971961E+07
0.5122420E+06 0.5971934E+07
0.5122440E+06 0.5971925E+07
0.5122720E+06 0.5971930E+07
0.5122770E+06 0.5971895E+07
0.5122720E+06 0.5971894E+07
0.5122250E+06 0.5971885E+07
0.5121700E+06 0.5971884E+07
END

9
0.5118040E+06 0.5971068E+07
0.5118470E+06 0.5971000E+07
0.5118070E+06 0.5970966E+07
0.5117430E+06 0.5970930E+07
0.5117200E+06 0.5970950E+07
0.5117090E+06 0.5970949E+07
0.5116780E+06 0.5970993E+07
0.5117120E+06 0.5971018E+07
0.5117230E+06 0.5971014E+07

0.5118040E+06 0.5971068E+07
END
END

This file type is the same as a file for points but for lines. The first number indicates the start of each individual line by giving it an unique ID (“9″ and “12″ are the unique IDs for the two lines). The rows of numbers after the ID represent x,y coordinates (in this case UTM coordinates – we will learn about this later) and the END statement defines the termination of each line as well as the completion of the layer (end of the field). These numbers can be brought into GIS software to make shapes (the campus buildings actually). Instead of generating these features we are going to use bear point data to create a line that expresses the movement of a bear.

Question 2: Last lecture we learned what the start and end points on a line are called as well as the points along the line. What feature types can be created for the file above (the generate file)? Explain your reasoning using the terms we learned in lecture.

Making lines from points

Load the bears95 from this week lab folder on the L: drive. You can also load the roads shape file from this weeks lab folder.

Make sure you have your toolbox panel active (viewable).

First thing you should always do is explore the dataset, understand what the attributes are. If we inspect some of the points in the bear layer we may guess as to the probability of there being one bear in several areas over a period of time. Pick out several points of interest to build a line from.

  • Add bears95 and roads layers – as shape files
  • zoom to an area of interest
  • use the select tool (Select menu –> select by rectangle)
  • Search for the tool “Points to Path” in the toolbox panel
  • order by “date_path”
  • group by “Activity_c”

You can play around with grouping and ordering the creation of the line as you see fit. You could even try to group by DATE and order by Date_path. Too bad we do not have the time of day the data was collected.

Make sure you have saved you output as a shapefile – otherwise you could lose your work.

You should also save your project (Project –> save project). As the semester moves along, you will realise that saving often is very important. Often students will save may versions (e.g. lab1a, lab1b…) that they loop through in order to take a step back if you did something crazy in your project.

What is the difference between saving your project and saving layers as shapefiles?

Digitizing – Points to Lines to Polygons

Start a new project.

As the use of GIS and related fields have become more common place, there has been a great amount of data created or collected. For instance the road layer we are using could have come from municipal CAD drawings or digitized directly using GIS. This means that at one time, all old data had to be digitized (drawn using digital means – a computer) by applying similar techniques as we are about to use.

Steps for digitizing

Using the bear95 layer we will trace some points approximating where a bear’s (or several bears) cruising ground is. We want this new layer to be clean so we need to apply the concepts of threshold and snapping.

  • Go to settings –> options –> digitizing
    • use map units for snapping at 15 metres
    • map units for search 20 metres
  • Add the snapping toolbar and from there you can pick the layers to snap to and customize the snapping options
  • Add a new “line” shape file layer (layer –> New –> New shapefile layer)
  • Check the projection to be UTM 10 (26910)
  • Start editing by applying the “yellow pencil” (can be done by right clicking the layer or hitting the menu button)
  • Draw some lines – three or four that connect (try to snap to bear location points – look for the red target).
  • Save your edits to your new layer

Turn them into polygons – Two Ways

  • Under the Vector menu use the “lines to polygon” function to create polygons.
  • Repeat the polygon building by using the polygonize tool (It can be found in the toolbox you used earlier)

Thinking about the difference between these 2 ways of making polygons

The first illustrates the lines that were digitized from following the point layers – (i.e. turning on the snapping options to make use of the bear point layer below our line that is digitized).

The lines were drawn in the following order:

  1. red (it is underneath the lines the yellow and orange lines share with it)
  2. yellow (it has common lines with red and grey)
  3. orange (sharing with red)
  4. grey (sharing with yellow)
  5. blue (is shares with no other lines and has only two vertices)

The second is the polygons created using the “lines to polygons” feature standard under the Vector menu.

The third illustrates the polygons created using the Polygonize Tool.

Question 3: Try to explain why the two polygon layers are different. Can you determine the process used to create the polygon features? (HINT: the standard tool follows the lines in the order they are drawn to see if they form a polygon, the polygonize tool reviews all lines to see what combinations can be used to form areas).

Create polygons directly

Using the skills we just learned, create a new polygon shapefile and hand digitize an area that you would want to investigate further – like an area of interest that you would focus the rest of your work on.

  • Create a new “polygon” shape file layer
  • Activate editing of that layer
  • Go to settings –> snapping options
    • Click to activate the bears95 and your new polygon layers
    • Set the snapping tolerances to 20 map units
    • Set to both “Vertex and line” in the snapping toolbar
  • Draw some polygons
    • Draw features without them touching
    • Draw features that are clean and touch
    • Draw features that overlap
    • Draw some holes in some feature (you will have to find your advanced editing tool bar)
  • Add vertices to features
  • Move vertices within features
  • Save your edits

Question 4: Submit (email) your shapefile of the digitized polygons so your TA can open it in their own QGIS project. Remember – how many files are there in your shapefile? Which ones do you need?

Adding your Geotagged photos to QGIS

In the tutorial this week, you should have created a KMZ or KML file from GPSPrune.

Open up a new project in QGIS and add in those photos points (instead of loading a shapefile from the L:\ drive, you are loading a KMZ file from your K:\ drive

Once that is loaded use the QucikMapServices to add a webmap base layers below the points to give geographical reference to your data.

Question 5: Submit (email) a screenshot of the geotagged photo points with the base map to your TA for review. You can use the snippet tool on osmotar (or something similar on you own computer).

Assignment #2 – 5% due next week

  • Save the answers for the questions 1, 2, 3, 5 in this lab as lastname_firstname_geog204_A2.
  • For question 4, compress all necessary files for your spatial layer (digitized polygons) and save it as zip file. Name the file as lastname_firstname_geog204_zip_A2. Or you can compress everything in one zip file
  • Submit both files by email to your TA at next week lab session
Categories: GEOG 204Labs