How did we do it?

Who did it?

Project management and website construction by webmaster and Lincoln University ecology lecturer, Jon Sullivan.

Scanning by Lincoln University summer scholar, Sean Fergus.

Proof-reading by Lincoln University PhD students, Chris Berry and Lora Peacock.

Database additions and editing by Lincoln University Associate Researcher, Laura Molles.

Special thanks to Owen Watson and Richard Bacon for advise.

Who funded it?
      $18k funding for this project, over the 2003-2004 financial year, was provided by the New Zealand Government through its Terrestrial and Freshwater Biodiversity Information System (TFBIS) Programme. Formed as part of the Government's committment to the NZ Biodiversity Strategy, this programme is committed to increasing the awareness of and access to fundamental data and information about biota and biodiversity.

Computer software and hardware

Computers: Apple PowerMac G4, iBook
Scanning: HP Scanjet 5550c scanner, with a 35-page automatic document feeder
Batch and manual cleaning of scans: Adobe Photoshop
Optical Character Recognition (OCR): ReadIris Pro 7.0, 9.0
Proof-reading of OCR files: Microsoft Word
PDF creation and compression: Adobe Acrobat and Distiller, Apple ColorSync (part of OSX 10.3)
PDF editing and figure insertion: Adobe Illustrator
Webpage creation: Bare Bones BBEdit
Database creation: Filemaker Pro
Server OS and software: LINUX, PHP, MySQL, CPanel, PHPAdmin

Summary of the process
      To ensure a high standard of accuracy with text recoginition, especially for ecological terminology and scientific names, we elected to scan all issues in-house, employing students at Lincoln University to assist with the project.

      Older NZ Journal of Ecology articles were scanned, OCR to fully formatted rich-text files, then proof-read. These files were converted into PDF files. Their figures were concurrently cleaned in Photoshop and GIF files of these figures inserted into the PDF files using Adobe Illustrator.

      Proceedings of the NZ Ecological Society issues had less reliable typesetting, and some of the glossy pages tended to slip in the automatic feeder of the scanner. Pages of all ProcNZES issues were cleaned in Photoshop (e.g., to remove the text from the other side of each page that came through on the scans), before using ReadIris Pro to create image PDFs with a hidden layer of OCR text (not proof-read). These PDF files were then compressed with Apple ColorSync.

      While the PDFs were produced, a database of citation data was created in Filemaker Pro, a popular, user-friendly desktop database program. The titles, abstracts, authors, etc. were all added to this database using a combination of copying-and-pasting from PDFs, typing in, and editing from existing databases. A MySQL database was created on the server with the necessary fields for the webpages, and the Filemaker database was uploaded into the MySQL database (for instructions on connecting FileMaker to MySQL, see http://www.fmpdev.com/fmp2mysql.html). MySQL is a popular, industrial strength database that runs on a web server and is free for non-commercial use.

      Webpages were written in HTML and PHP using BBEdit. PHP is an open-source programming language designed for creating dynamic websites. By using PHP to call on the contents of the MySQL database, about ten webpages had to be coded to dynamically create all the >1000 NZJE webpages accessible by the user.

Details of the process
      We are converting our notes into an online manual detailing the steps we took to create this site and the PDF files. This will be available shortly. Meanwhile, the scanning process is detailed in the appendix of Sean Fergus's summer scholarship final report, which can be downloaded here (1.8 MB PDF file). The opinions expressed in this report are Sean's alone, and do not necessarily represent the NZES position on electronic publication of research.