Home :: Developers

COWS Web Processing Service (COWS WPS)

«  COWS WPS   ::   Contents   ::   2. Installation  »

1. Introduction

1.1. A brief overview of COWS WPS

1.1.1. COWS WPS Overview

The COWS Web Processing Service (WPS) is a “generic” web service and offline processing tool developed within the Centre for Environmental Data Archival (CEDA). The CEDA OGC web services (COWS) is a set of Python libraries that allow rapid development and deployment of geospatial web applications and services built around the standards managed by the Open Geospatial Consortium [OGC].

COWS WPS grew out of a need to provide a flexible framework to deploy ad-hoc web services that did not fit into the scope of the commonly deployed Web Mapping Service [OGC-WMS] and Web Coverage Service [OGC-WCS]. Whilst the aim of the COWS WPS is conformance with the WPS version 1.0.0 standard it has been used within CEDA (including the UK Climate Projections User Interface [UKCP09-UI]) to support operational systems that require asynchronous processing, job scheduling and management of user jobs.

Rather than developing separate Web Services (and interfaces) for each project and requirement the COWS WPS provides the opportunity to deploy “processes” inside a single framework. The framework itself contains a set of utilities that the administrator can call upon such as zipping up output files, automatically generating an HTML submission form, polling offline job progress and e-mailing users when a job has completed.

The documentation presented here is intended primarily to suit the needs of administrators and should developers. It should also be useful to the user who is accessing the COWS WPS via the web, or using the WPS user interface.

1.1.2. The CEDA OGC Web Services libraries (COWS)

CEDA OGC Web Services [COWS] is a Python software framework for implementing Open Geospatial Consortium web service standards. COWS emphasises rapid service development by providing a lightweight layer of OGC web service logic on top of Pylons [Pylons], a mature web application framework for the Python language. This approach provides developers with a flexible web service development environment without compromising access to the full range of web application tools and patterns: Model-View-Controller paradigm, XML templating, Object-Relational-Mapper integration and authentication/authorisation. COWS contains pre-configured implementations of WMS, WCS and WFS services, a web client and WPS.

1.1.3. Web Processing Service (WPS)

The OGC Web Processing Service version 1.0.0 specification provides a means of wrapping up arbitrary processes (such as GetAPlot or RunAModel) in a common framework with the following features:

  • web service interface, using POST or GET.
  • asynchronous reporting and control of jobs.
  • a defined XML interface for responses, including exceptions.
  • a common format for passing arguments to the server.
  • job status interrogation.

The COWS WPS extends this functionality to include:

  • Inform users when a job has completed (or failed).
  • Provide a configuration system that allows new processes to be added via a simple configuration file and a single Python interface module. This feature is only available to the WPS administrator in the current implementation and new processes are only identified after the service is restarted. However, the “plug-in” methodology could be extended to allow authorised users to add processes at runtime.
  • Connection to a scheduling tool that communicates with external offline processing nodes. The SOA layer and the Offline Processing layer share a file system which allows either layer to read/write status information to a single location.
  • A multiple-processor server application that can run “quick” tasks within a worker pool of connected processes.
  • Make a “costonly” (or “dry-run”) request before submitting a job that results in an estimate of the response size and duration without actually executing the job. This is implemented as an additional argument to the top-level WPS Execute Request so that it is available for use with all processes.
  • Report a job history for the entire system or the individual user.
  • Zip up groups of output files and report details of their contents in the XML response.

This functionality enables selecting and viewing of thumbnail plots within the client tools.

A specific feature of the system that required a great deal of integration between the PHP/JavaScript UI and the Python WPS was the tracking of user jobs via the UI Jobs page. The WPS specification provided a mechanism for presenting a persistent URL that returns the status of a given job. The UI employed AJAX technology to poll this “status URL”, parsing the XML and displaying a table of outputs that can be interrogated by the user. For accessing information about previous jobs, an additional process was deployed under the COWS-WPS that returns an XML-encoding of the metadata for previous jobs issued to the WPS by a given user. This extension of the WPS capability is likely to be an essential requirement of any implementation that includes asynchronous job management.

The COWS WPS provides the standard Web Processing Service methods as web-accessible end-points:

GetCapabilities
an XML doc saying the names of processes deployed within the WPS
DescribeProcess
details on the inputs/outputs of a given process
Execute
a request to execute a job using a providing the process ID and input parameters

Each “process” can be thought of as a Web Service that has is deployed within the WPS container. Rather than discovering and calling the API of a bespoke Web Service the client calls the API of the WPS with a set of arguments relevant to the “process”. The advantages of this approach are that a single framework can manage multiple Web Services and work-flows, with all common functionality being housed outside the process. Such functionality includes:

  • scheduling of off-line jobs and asynchronous polling
  • zipping of output files and reporting to the user on completion/failure of a job
  • auto-generation of HTML forms to submit jobs
  • application of a common security layer
One view on WPS “processes” is to think of them as the equivalent of command-line scripts with input arguments. Hence it is easy to wrap any old coded function within a WPS process as long as the configuration file provides appropriate information about the input/output arguments and their data types.

1.1.4. History of the COWS WPS

Catering for jobs that would run for different durations and produce a range of different outputs required a solution that managed both in-process execution of processes and asynchronous job management. When selecting a tool or framework to deliver this flexibility the OGC Web Processing Service (WPS) specification was considered a suitable candidate. Whilst still in its infancy, the WPS interface provides a means of wrapping up arbitrary processes (such as GetAPlot or RunAModel) in a common framework with the following advantages:

  • Web Service interface, using POST or GET.
  • asynchronous reporting and control of jobs.
  • a defined XML interface for responses, including exceptions.
  • a common format for passing arguments to the server.
  • job status interrogation.

1.2. COWS WPS and standards-compliance

1.2.1. WPS 1.0.0 Specification

The COWS WPS was initially developed to meet the draft WPS standard at version 0.4.0 but has been updated in light of the publication of the WPS 1.0.0 specification [OGC-WPS].

1.2.1.1. Application Profiles

WPS 1.0.0 introduces the concept of Application Profiles as means of providing and publishing domain-specific processes to aid interoperability in terms of building clients and utilising the publish/find/bind paradigm. In the case of this project the outputs are highly tailored so in most cases the processes are unlikely to be generally applicable to other climate-related datasets. However, the development of Application Profiles for more general climate-plotting processes would be beneficial as there are common requirements for plotting tools outside the 2-dimensional view covered by WMS.

1.2.1.2. Web Service Description Language and SOAP bindings

Version 1.0.0 of the specification describes how to implement a WSDL (Web Service Description Language) and SOAP interfaces to WPS. At present this type of interface is not implemented in the COWS WPS. We anticipate the development of this capability in future.

1.2.1.3. Compliance issues

The COWS WPS is aiming for standards compliance with WPS 1.0.0. Through interoperability testing with other WPS server and client implemetations we anticipate achieving a high level of standards compliance.

1.2.1.4. Web Server Gateway Interface (WSGI)

The implementation of the Python-based COWS WPS uses the Web Server Gateway Interface [WSGI]. This standard is allows multiple web-applications to be layered and chained in a service stack due to their conformance to a common environment and interface. The WSGI layer is implemented using Python Paste and Pylons technologies.

1.3. About this document

The reader is asked to note the following regarding this document:

Is the documentation up-to-date?
Documenting a framework as complex as the COWS WPS takes time. It is likely that from time-to-time there will be aspects of the documentation that are missing and/or inconsistent with the code-base. If you would like to contribute to the documentation you are very welcome and any error reports are very welcome.
Conventions for representing source code
When file/directory paths are presented without being prefixed by a “/” character the assumption is that the path is relative to the top-level directory of the source distribution. This is the directory containing the WPS application configuration file.

«  COWS WPS   ::   Contents   ::   2. Installation  »