Home :: Developers

COWS Web Processing Service (COWS WPS)

«  2. Installation   ::   Contents   ::   4. WPS processes  »

3. Main features

This section aims to highlight some of features of the COWS WPS that make it a useful and flexible framework.

3.1. Built on Python web technologies

Python is a scripting language with a clear and readable syntax. It is becoming increasingly popular with the scientific, commercial and web communities because of the range of its portability, extensive range of supported libraries, tools and frameworks, human readability, fast-prototyping capabilities and all round greatness!

The COWS WPS is able to make use of a selection of Python tools that make light work of delivering robust web services. These technologies include the Pylons framework for web development, Numpy and CDMS (for scientific data handling) and matplotlib (for visualisation).

The COWS WPS User Interface (CWUI) employs the open source JQuery Javascript framework to support the delivery of the web front-end.

3.2. Common framework for Web Service deployment

The main driver for developing the COWS WPS was the need to produce a framework that provided:

  • A generic framework for deploying web services (as “processes”)

  • A process configuration wrapper to web-enable existing useful code (in any language)

  • A scalable system for offline processing

  • The ability to calculate and provide information on the “cost” of a job prior to execution

  • Information about previous jobs and their outputs

  • A method of recording and polling the status of offline jobs

  • A set of common utilities such as:

    • Zipping of output files
    • E-mailing job completion/failure to user

3.3. Asynchronous processing and scalability

One aspect of the COWS WPS that makes it useful for large-scale processing is the connection to the SGE tool that manages a set of job queues that can execute jobs on offline processing servers. Any number of identical servers can be deployed and the system can be readily scaled up or down depending on the requirements. The asynchronous job management in the WPS specification provides a standardised method of requesting information about the job status which can be employed within any client application calling the WPS.

3.4. Dry-run execution for resource estimation

Some jobs will take 2 seconds to run and will return 50 bytes of data. Others may run for 2 days and return 50 Gigabytes of data. The COWS WPS is able to manage both types of request and is also equipped with an additional feature for “large” requests. The user/client may request a “dry-run” before actually executing a job. This dry-run will return the “cost” of the job in terms of its estimated duration and total volume of outputs. The advantage of this approach is that it allows:

  • the WPS to (politely) refuse certain requests that exceed any resource limits set in the configuration details
  • the user can ask the WPS about the “cost” of the request to help decide if the request is appropriate

3.5. WPS User Interface

As the code has evolved the COWS WPS has been extended to include a web front-end. The COWS WPS User Interface (see The COWS WPS User Interface (CWUI)) provides the following:

  • automated generation of submission forms for new processes
  • test-client for the WPS services
  • browser front-end to job/output management functions
  • login provider for secured processes
  • administrator tools for job/user management
  • a development tool for pushing the capabilities of WPS

The automated generation of submission forms for processes adds to the set of generic features to support ad hoc web services described above. The deployment of a new process is rapid, and immediately provides both server and client application.

3.6. Secured services

Integration with the CEDA Security client stack enables the COWS WPS to secure individual processes based on the URL request. This allows fine-grained control of the access control policy based on the roles assigned to a given user. The WPS does not actually manage the securing of the services but is able to interrogate the user credentials for required information about the “roles” attributed to a given user.

3.7. Useful processes - and add your own

The COWS WPS framework requires an investment of time to set up and get running. In some cases you may wish to create a set of Virtual Machine builds in order to deploy the WPS. Once installed and deployed, there are a set of Supported processes that come “free”. These include data extraction tools for datasets described in the appropriate formats, a WCS-wrapper process and a tool for “dumping” the metadata from a NetCDF file.

Whilst the supported processes may be of use, the real power of the COWS WPS comes in deploying your own Local Processes. The next section (WPS processes) describes how you can deploy your own processes under the WPS based on a template Python wrapper module and a template configuration file.

«  2. Installation   ::   Contents   ::   4. WPS processes  »