How do we count animals? How do we keep our data? How do we disseminate our data? At what cost? And why...? There is a lot of questions related to wildlife management in Tanzania. Some are technical, and some bordering to the more political sides of wildlife management. All of them bear witness of the complexity of the ecosystems we are dealing with.
Counting animals is about generalisation. It is about reducing each individual to a series of numbers. Age, position, weight, values from blood samples, behaviour and even... stripes. With maps we can, for better or worse, take this even further. With the proper analysis based on generalisations we can see patterns of birth, disease and ultimately death. For the geographer, spatial patterns emerging from maps, can be the tools of understanding. For the biologist, blood samples can be means for the same understanding. Combining forces we can arrive to new conclusions, or even new questions.
During the last two years I have worked on a system for storing, calculating and presenting survey data for the Tanzanian government. This article is about parts of the work we are doing at Tanzania Wildlife Research Institute (TAWIRI). As an intern financed by the Norwegian FK in collaboration with NTNU and NINA I have been given an opportunity to learn about animal surveys by the experts at TAWIRI. It has also given me an opportunity to contribute using my own skills as a geographer and programmer.
In my view one of the main obstacles in this work has been the lack of standards for exchange of data. As a consequence we have started working on formalising the use of Quarter Degree Grid Cells as a means of sharing survey data. Developing actual tools has been an integral part of this work. In this article I will examine the possibilities of using web mapping in the service of biodiversity research and management.
The illustrative project of this work is the Tanzania Mammal Atlas Project (TMAP), currently hosted at the Carnivore centre, TAWIRI, Arusha.
Please feel free to send me emails with corrections and suggestions of any kind regarding this article.
Read on by using the below buttons.
...page...
Contents
Introduction
A web mapping systems most central features is to render spatial data (maps, images, and vector data) for the web. More complicated systems might make available tools for data editing and analysis.
A basic web mapping system can be made by using plain HTML. Given a map-file and a few lines of code one can easily set up links from the map-picture and as such, give the user a spatial tool - a web map. So - knowing html would quite fast put you in the front seat of your very own, but very basic web mapping system.
For all practical purposes you would probably be better off using one of the current systems available. A lot of effort has gone into making these web mapping systems both flexible and versatile.
Such systems would usually cover tasks like:
- displaying shapefiles
- conditional display certain shapefile elements (size/color of roads, etc)
- convert between different formats
- giving access to databases for further conditional manipulation
- editing of data
- annotations of data
- complicated analysis (eg fire controll analysis, network analysis, etc)
In the next chapter current systems for web/mapping will be given a short presentation. I will then proceed to do a more in/depth presentation of web-mapping with MapServer? used to present TMAP data.
Current web mapping systems
Web mapping systems is a growing industry. Moving maps from the desktop to the browser of potentially thousands of users is something a lot of us want. The drive is cost of purchasing the desktop based systems, as well as the costs of ownership - being time and money to learn to use the desktop based systems. Properly designed web mapping systems has the potentialof making everyday life both cheaper and easier for the end user.Both commercial solutions and Open Source based solutions are available. The systems available span from simple presentation tools to interactive systems which looks very much like your familiar desktop based GIS systems.
Keeping track of all of them and being able to produce a balanced test of them is not within the goals of this article. I should probably remind the reader that my intention is to demonstrate how putting your data online could be done with a minimum of time and cost. This would exclude most commercial systems.
I still would like to mention the most interesting and used systems here. In the next paragraphs I will in short present MapServer?, Arc Internet Mapping Server and list some other systems.
MapServer
MapServer? was developed by the University of Minnesota in cooperation with NASA and the Minnesota Department of Natural Resources (MNDNR).The list of specifications as described on the Mapserver homepage is:
- Advanced cartographic output
- Scale dependent feature drawing and application execution
- Feature labeling including label collision mediation
- Fully customizable, template driven output
- TrueType fonts
- Map element automation (scalebar, reference map, and legend)
- Thematic mapping using logical- or regular expression-based classes
- Support for popular scripting and development environments
- PHP, Python, Perl, Ruby, Java, and C#
- Cross-platform support
- Linux, Windows, Mac OS X, Solaris, and more
- A multitude of raster and vector data formats
- TIFF/GeoTIFF, EPPL7, and many others via GDAL
- ESRI shapfiles, PostGIS, ESRI ArcSDE, Oracle Spatial, MySQL and many others via OGR
- Open Geospatial Consortium (OGC) web specifications
- WMS (client/server), non-transactional WFS (client/server), WMC, WCS, Filter Encoding, SLD, GML, SOS
- Map projection support
- On-the-fly map projection with 1000s of projections through the Proj.4 library
Arc Internet Mapping Server
ESRI describes Arc IMS themselves this way "ArcIMS is the solution for delivering dynamic maps and GIS data and services via the Web. It provides a highly scalable framework for GIS Web publishing that meets the needs of corporate Intranets and demands of worldwide Internet access. ArcIMS services can be used by a wide range of clients including custom Web applications, the ArcGIS Desktop, and mobile and wireless devices. Using ArcIMS, city and local governments, businesses, and other organizations worldwide publish, discover, and share geospatial information."All of this is true. ArcIMS provides a host of possibilities for implementations and an equal availability of training and other related events. It should also be mentioned that ESRI has a well developed network of worldwide distributors. In East Arica the distributor is Oakar Services Ltd placed in Kenya.
So what is the cost for using ArcIMS? Entry fee for ArcIMS is around $8.000-10.000 In addition you must expect to pay a yearly fee of 20% ($1.600-$2.000). This is for one processor only. If you expect a lot of traffic on your website one processor might not be enough.
Some of the criticism on ArcIMS is that it is expensive. With the above mentioned entry fee you would still have to pay developers and complementary software like Microsoft Server 2003, a commercial database solution (Oracle or MSSQL).
Slow? I really never got around to try it out. I would love to compare MapServer? and ArcIMS, but having more knowledge on the use of MapServer? it could very well lead to unfair conclusions. All I could find were some comparisons on the net:
- ArcIMS vs. MapServer - Some Hard Data (from The Programming Consultant Newsletter)
- A Comparison of ArcIMS to MapServer (Powerpoint presentation, 307 Kb)
Google maps
This is a system existing a bit on the side of what I am presenting in this article. Google Maps should however be noted for their online maps with strong tools (API) for interacting with the maps. Programmers can produce good maps with a high degree of flexibility using Google Maps. More information is available here:Other systems
Other web mapping systems I have managed to find on the net were:- MapMatrix (Heavy Microsoft server platform integration. Low price $399, and SDK for free)
- Maptitude for the web (.NET/Visual Basic/ASP - ie. heavily integrated with the Windows platform)
- CMAP("CMap is a cross map API designed to make coding and switching between online maps easier. It is currently in its initial release and works well for what it does. Currently we support Google Maps, Yahoo Maps, and Google Earth KML files.")
But in this article I will focus on MapServer? and show how it is possible to build a biodiversity website using MySQL and PHP. So - for better or worse - ArcGIS is out of reach. But then again - why bother. Mapserver is free - and its good!
Now let's move on to the fun stuff! I will start with a snapshot of MapServer? functionality.
Basic Mapserver concepts
Bill Kropla, author of the book Beginning MapServer? puts it quite to the point when he explains what Mapserver is all about:"Mapserver creates map images from spatial information stored in digital format."
The long story goes...
MapServer? is a computer program which works on top of a web server. MapServer? can be made accessible using just plain HTML, PHP, Perl etc... Common for all modes is the use of a template file. This file, a .map-file, is a setup file for the map which is about to be made. The file points to the relevant shapefiles, projections, layers, labels, scale bars, reference maps, etc
I have made a small example which shows Tanzania based on a couple of familiar shapefiles, namely the Tanzanian political border and the Tanzanian waterbodies. In this example I will use the following:
- .shp-files (tz-politicalboundary.shp and tz-waterbodies.shp)
- .map-file with definitions of the map presentation
- .php which pulls the above together.
Let's start with the .php-file.
The .php-file initiates a map object and requests the .map file about what information should be included in the map. This is what the .map-file looks like:
Combined this produces a simple map with country borders zoomed in on Tanzania. Note how the.map-files starts with general definitions and file home directories. Since we are presenting two layers we accordingly have two LAYER-objects. It is quite straightforward.
The resulting map is the one below. It is a basic map with no possibilities for zooming, panning or similar. But with half an hours more work I guess it would be possible to do both zooming and panning. Then add a few more layers like roads, forests, names of towns and we'd have a nice Tanzania web map on or server.
Now, what would the use of this be? Why not make the map using ArcGIS? Well, the whole point with web-mapping is to let the user get access to your data, and even let him/her make some individual choices. Maybe the user wants to zoom in, pan, or select just some of the layers you present to them? Perhaps most central to web-mapping is the possibility to give the user access to "fresh" datasets combined to maps.
With MapServer? in combination with PHP you can control the final map. In a form you may give the user the option to deselect some layers, pan to any corner of the world, zoom in/out, and so on. It is fairly simple, and we will do some of this in the TMAP web mapping application.
Tanzania Mammal Atlas Project - their needs
TMAP at the Tanzania Wildlife Research Institute needed an upgrade of their database systems. I usually work with the survey information systems at TAWIRI, but all things considered we decided to focus on their needs for a week or so. As one of the stakeholders in the TMAP project, it is also TAWIRI's part of the agreement to support TMAP in developing their database systems.TMAP wants to make maps on presence of certain mammals in Tanzania. Their goals are to establish baseline data for many of the Tanzanian Mammals. This, in turn, will facilitate future work with an action plan for Tanzanian mammals.
To get a hold of all this data one needs to get sightings data from the general public. Publishing work-in-progress maps and top contributor lists is one way of building a community and gaining momentum in such a process. Except for this, TMAP are also relying on camera traps, historical records (books) and museum collections - to mention some of their methods.
Initial system
Initially TMAP is based on previous data-sets from "The Tanzania Carnivore Atlas Project" and "Cheetah and Wild Dog Watch Campaign". The data was stored in an Microsoft Access database, and at regular intervals queries were run to produce .dbf-files for ArcGIS consumption. This process unfortunately involved a lot of steps to produce even the simplest maps. The manual process of producing maps for all the involved species led to a sbstantial time-lag from data entry to the data presentation.The first part in this chain is MicrosoftAccess?.Microsoft Access is in many ways a very nice database for starting up a project. Making forms for editing and adding data is easy. Reportmaking is also fast. Making queries by doing visual edits has given me the power to make extensive queries which are not normally within my reach. But there were some issues:
- Copying and pasting between tables (+9000 posts)
- Speed issues with large tables
- Webpage integration
- Export of tables (Visual Basic programming might be needed)
- Multiuser support is not good - this is after all a desktop database
All in all the existing database was not good enough, and we decided to move to MySQL. The main reasons were it is convenient (I know how to work with MySQL), it scales very well, and it is free.
The process of manually making 70-80 maps using ArcGIS every third month is in itself time consuming. If you remember how you did it. If you didn't have to reinstall your computer last week. If your GIS expert didn't move on to greener pastures. If the computer didn't have a Windows(tm) breakdown. If your ArcGIS license didn't expire. And so on...
The Initial system as described here works well at Tanzania Mammal Atlas Project. With competent personell and good equipment it is functional, albeit a bit cumbersome. TMAP and me wanted to look at other solutions.
The renewed system
I am a fond believer of iterative development, and this system (as well as SISTA at TAWIRI) are a good examples of how it is possible to move forward without redoing all the work.The core of this web mapping project is to produce maps which gives us an indication on where we can find observations of a particular mammal. To do this we need an understanding of which data are entered, what Quarter Degree Grid Cells (QDGC) are, how observational data can be assigned a particular QDGC. Furthermore we need to know how background shapefiles (country borders, park borders, rivers etc) and QDGC shapefiles are used to produce maps where QDGC with observations are contrasted to QDGC without observations. Designing good maps is also a part of web mapping.
In this section I will go through five important parts of the system:
- Data structure
- MySQL - our new database system
- QDGC - which is the generalisation level we are working with
- MS Access - our database entry system
- Web-pages - presentation of general informationregistered in the database
The MapServer? and map production will be given special focus in the next chapter.
Data Structure
Upgrading from MS Access was not only a matter of moving the database files to a new database framework. Part of the work done was also to restructure the database. The technical term for this restructuring is database normalisation. The process promotes the logical consistency of stored data. One example is the names of the observers. Instead of storing the names in the table containing the observations the names are replaced with a numerical reference.The main table in the TMAP database consists of the following columns:
| Column name | Examplevalue | Description |
| id | 5431 | unique reference number for this observation |
| ref_specie | 7 | Reference to specie table (7=lion) |
| lat | -2.644160000000000 | latitude value (dd.nnnnn) |
| lon | 35.24922997 | longitude value (dd.nnnnn) |
| obs_date_day | 12 | day of month |
| obs_date_month | 7 | date of month |
| obs_date_year | 2006 | date of month |
| obs_exacttime | 21:07 | exact time of day |
| no_ind | 3 | number f individuals |
| no_juv | 2 | number of juveniles of the above total if applicable |
| no_alive | 3 | All of them alive |
| notes_habitat | 'Near kopje' | Notes on the habitat |
| notes_general | 'Good observation by lights from car' | Generalnotes |
| date_data_entry | 18.08.2006 | Date |
| date_data_received | 01.07.2006 | Date |
| qdgc_1 | E035S02C | QDGC Level 1 in this example |
| ref_observer | 34 | Person registeredwith umber 34 in tbl_person |
| ref_observation_timeofday | 4 | Alternatives like mid-day, morning, evening, etc... |
| ref_observation_submissionmethod | 5 | Alternatives like email, telephone, etc... |
| ref_region | 7 | Reference to table tbl_region |
| ref_reliability | 9 | Reference to reliability like Good, Very good, etc... |
| ref_observer_condition | 12 | Condition |
| ref_habitat | 11 | Woodlands, Grassland, etc... |
| ref_observation_frequency | 3 | Wether the person had a longer or shorter stay in the area |
| ref_observation_type | 4 | Phototrap, TAWIRI survey, etc |
| ref_publication | 2 | Relevant if the observation is from a book |
| publ_page | 223 | Page referred if publication is given.. |
| last_update | 21.09.2006 15:22:02 | Always when data in the table is entered a new stamp is given |
MySQL database
A MySQL database is simply a database. It doesen't boast nice form tools or report engines. You communicate with it by sending SQL queries and it returns an answer in a tabular format. It has a lot of features, and as opposed to MS Access it is very fast. MySQL comes with administrative tools which facilitates for the communication with the database. But you could also open up a command line box and log in.In the above example I requested a list consisting of all the QDGC in combination with the common name of the observed species:
SELECT tbl_observation.qdgc_1, tbl_species.name_common
FROM tbl_observation
INNER JOIN tbl_species
ON tbl_observation.ref_species = tbl_species.id;SQL code which counts the number of observations of a given specie for a given QDGC is neither a very complicated query:
SELECT count(id) FROM tbl_observation WHERE qdgc_1 = 'E035S03A' AND ref_species = 7
This particular square had 45 observations of lions (arbitrary code for lion in the database is 7).
QDGC
Quarter Degree Grid Cells (or QDS - Quarter degree Squares) are a way of dividing the longitude/latitude degree square cells into smaller squares. QDGC represents a way of making (almost) equal area squares covering a specific area to represent specific qualities of the area covered. The squares themselves are based on the degree squares covering earth. Around the equator we have 360 longitudal lines lines, and from the north to the south pole we have 180 latitudal lines. Together this gives us 64800 segments or tiles covering earth. The form of the squares becomes more rectangular the longer north we come. At the poles they are not square or even rectangular at all, but end up in elongated triangles.Each degree square is designated by a full reference to the main degree square. S01E010? is a reference to a square in Tanzania. 'S' means the square is south of equator, and 'E' means it is East of the zero meridian. The numbers refeer to longitudal and latitudal degree.
A square with no sublevel reference is also called QDGC level 0. The QDGC level 0 squares are themselves divided into four.
| A | B | |
| C | D | |
To get smaller squares the above squares are again divided in four - giving us a total of 16 squares within a degree square. The names for the new level of squares are named the same way. The full reference of a square could then be:
- S01E010AD
The number of squares for each QDGC level can be calculated with this formula:
number of squares = 2**(2d)
(where d is QDGC level)
Table showing level, number of squares and an example reference:
| Level | Squares | Example |
| 0 | 1 | S01E010 |
| 1 | 4 | S01E010A |
| 2 | 16 | S01E010AD |
| 3 | 64 | S01E010ADC |
| 4 | 256 | S01E010ADCB |
| 5 | 1024 | S01E010ADCBD |
| 6 | 4096 | S01E010ADCDBA |
Based on this Julian Blanc helped out in generating QDGC shapefiles with the following attributes:
| COL_ROW | 1 - 22 | |
| COL | 01 | |
| ROW | 22 | |
| SQUARE | E02911C | |
| LON | 29,25 | |
| LAT | -11,75 | |
| AREA | 3014,439149 | |
The files related to QDGC are available here:
The routine used to establish the QDGC-SQUARE value in the shapefiles is the same one which is used to establish QDGC value for observations in TMAP. The code is written in PHP and is freely available here:
MS Access front-end
To be able to put data into the MySQL database a MS Access front-end shell was built. The MS Access front-end looks suspiciously like the old MS Access database. This is no mistake. The application opens and seemingly works like it used to. If you didn't know the data were stored on a server you could easily come to believe that the data was still stored in the MS Access database system.
This is more evident when presenting one of the main forms:
MS Access is now effectively a MySQL client. All entries end up in the MySQL database. And all viewable data comes from the MySQL database. Since the data is stored on a server one can use several MS Access clients simultanously. This gives room for a more effective data entry process.
I would prefer to replace Ms Access with data entry forms on a web-page. MS Access requiers you or your organisation to own a license. With a little work one could easily build the web-forms. But in june/july I was just too busy.
Web pages
The web pages are based on the PHP preprocessor running on top of an Apache web server. A preprocessor like PHP let you build logic into your web-pages.This opens up for functionality like access controll, user controlled layout, reports and more.
A list of species is also available. Note the icon which is a link to a species specific map:
But PHP in itself can not make any maps. It would need a module which can handle the complex issues of shapefiles. That is where the MapServer? module becomes relevant.
Webmapping comes to TMAP
TMAP has decided to make their maps based on QDGC. Data is collected in lon/lat values, but for presentational issues the QDGC square is calculated and used for presentational issues. The use of QDGC is widespread, and in African atlassing it certainly is no novelty.QDGC used to be calculated manually, but the web pages have functionality to calculate QDGC based on longitude and latitude values. The QDGC column in the observational table is then updated. Consequently any observation in the database then has a reference to one QDGC.
Shapefiles
The shapefiles we are using are no different from the ones we are using on a daily basis in ArcGIS. In the TMAP project weare using:- the big georeferenced wsiearth.tiff-file
- the national parks shapefile
- Tanzania border shapefile
- protected area shapefile
- QDGC level 1 file
Finally producing the maps
The object of the TMAP project is to keep information about where certain mammals are in Tanzania. The TMAP database has one column with a reference to a QDGC. This reference is cross-checked with the QDGC level 1 file. Using an map-file and doing some PHP-magic soon produces the following map:To provide a full testing facility I have set up a testing website here:
Surely the system can be further developed. The layout of the maps could be better. Better features for zooming and panning could be implemented. Support for mulitiple QDGC levels is something I hope to be able to look at.
For a closer look at the distribution of pink elephants in Tanzania you can go directly to this webpage:
The code behind this demo project is available. Tor run the code you need to install MapServer?. You will also need the relevant shapefiles. Here you are:
The license is GNU General Public License. This means that you are free to use, install and further develop the software. If you do upgrades of the code, please inform (although you are not obliged to do so). This will make it easier to further develop the system.
External presentation
By setting up MySQL to mirror the contents of the TMAP database, it would be fairly easy to publish updated information from the database. The additional cost would be to have access to a server with MapServer? installed, running PHP and MySQL:Further work
In the wishful thinking part of this article my foremost wish would be that we through using QDGC could establish exchange formats for survey observations. Most of us has some needs for observational data. To compare data we do not always need 100% acurate positional data. QDGC could be the vehichle for this kind of data exchange. I know a lot of data owners based on practical, project policy rules, legal bindings, data ownership issues or concerns with regard to species protection choose not to share data. Sometimes we can work around those issues by generalising our data. Would anyone like to work on this together with me? Karibuni!Conclusion
I hope this article has shown that it is possible to build up an (internal) website for mapping biodiversity using Open Source Software. We do not need flashy expensive software to build websites supporting both ends and means for conservation work. I hope it also has shown that tools for internal consumption easily can be used to give the general public some access to our data. The consequenses of making generalised biodiversity data is an issue which should be discussed on a policy level, and it has not been an issue in this article.Thanks to...
- Blanc, Julian: Invaluable help the Quarter Degree Grid Cell systems
- Frederick, Howard: For inspiration to work with animal surveys
- Norwegian FK for funding
- TAWIRI: Dr Simon Mduma, Mr Honori Maiti, Mr John Kaaya and Mr Mwita Machoke.
- TMAP Project: Project leads: Dr. Sarah Durant with the Institute of Zooology, Zoological Society of London, and Dr Charles Foley whom also is with the Zoological Society of London. Other persons working on the project are Lara Foley, Alex Lobora and Edwin Konzo.
- Pedersen, Øyvind: Server hosting and a lot of general computer help.
- Rudsar, Synne: Proofreading
- Prof. Røskaft, Eivin: Lots of help in general
References
Baker, Neil. Personal communication, August 2005.Carswell et al. 2005. The Bird Atlas of Uganda. British Ornithologists Club.
Norton-Griffiths?, M. (1978). Counting animals.
Kropla, Bill. 2005. Beginning MapServer: Open Source GIS Development.
Links
- ESRI (ArcView/ArcGIS) - http://www.esri.com/arcgis/
- Mapserver - mapserver.gis.umn.edu/
- Microsoft Access - www.microsoft.com/access/
- MySQL - mysql.com
- PHP - www.php.net
- TAWIRI - Tanzania Wildlife Research Institute - www.tawiri.org
- ZSL - Zoological Society of London - www.zsl.org/
- WCS - Wildlife Conservation Society - www.wcs.org