Overview

This tutorial introduces the concept of tasks in HIPE and teaches you how to create a simple Jython tasks and import it in HIPE.

Audience

Read this tutorial if you have, or want to write, Jython functions that you want to turn into tasks. You do not have to turn your functions into tasks to use and share them. However, tasks have advantages (described in What is a HIPE task? below) that may be of interest to you.
All the algorithms used by data processing pipelines are written as tasks, in Jython (mainly PACS) or Java. If you plan to modify any of these, you should get familiar with tasks by reading this tutorial and the corresponding Java one.

Prerequisites

This tutorial assumes that you are familiar with basic Jython syntax and scripting. If this is not the case, read Chapter 1 of the Scripting Guide.

Objectives

This tutorial will teach you the following:
  • What a HIPE task is.
  • How to write a simple Jython task.
  • How to import and run the task into HIPE
  • How to create a HIPE plug-in containing the task.
  • How to share your task with other users.

Contents

What is a HIPE task?

A HIPE task is everything you see in the Tasks view of HIPE.
TasksView.png
Double click the All folder in the Tasks view to open it. Then type the first letters of the task name (for example, getObservation) to quickly jump to the task. Double click on the task entry to open its graphical interface in the Editor view.

Screen Shot 2014-02-24 at 14.19.38.png
We can make the following observations about a HIPE task:
  • A task does a specific operation. Hover your mouse pointer on the getObs entry in the Tasks view. A tooltip appears with a description of what this task does (in this case, loading an observation into HIPE).
Screen Shot 2014-02-24 at 14.21.03.png
  • A task takes input parameters. In the Inputs section of the getObservation interface you can see two input parameters, path and obsid. The path parameter is optional, while obsid is mandatory. Hover your mouse pointer on the parameter names to see tooltips with longer descriptions. Note also that each parameter is of a specific type (for instance, path is of type String).
TaskParametersTooltip.png

  • A task returns output parameters. In the Outputs section of the getObservation interface you can see the task output parameter. The value of this parameter is the name of the variable that will contain the result of the task execution. Note that in this case the parameter has a pre-filled default value, namely obs. This can also happen with input parameters.
  • A task has links to more information. At the bottom of the getObservation interface you can click Source to examine the source code of the task. For this button to be enabled you must have ticked the Would you like to have the source code installed? checkbox when installing HIPE. Click Help to view the task entry in the User's Reference Manual.
HIPE tasks can be written in Jython and Java. Jython is the language you use to write scripts in the Editor view of HIPE. Java is the language most of HIPE itself is written in. Writing tasks in Jython should be all you need for your data processing needs.

A simple HIPE task

The following is a simple Jython task that takes a table dataset as input and outputs an array with the averages of each row of the table dataset. For more information about table datasets, see the Scripting Guide.
class TableAverageTask(Task):  # 1
    'Averages the rows of a Table Dataset.'
    def __init__(self, name="tableAverage"):
        Task.__init__(self, name)
        self.setDescription("Computes the average of each row of a Table Dataset.")  # 2
        p = TaskParameter("table", valueType = TableDataset, type = IN, mandatory = 1)  # 3
        p.description = "The table of whose rows to compute the average"  # 4
        self.addTaskParameter(p)
        p = TaskParameter("average", valueType = Double1d, type = OUT)  # 5
        p.description = "The array of averages of the table's rows"
        self.addTaskParameter(p)
    def execute(self):  # 6
        columns = self.table.columnCount
        self.average = Double1d(table.rowCount)
        for column in range(columns):
            self.average.add(table.getColumn(column).data)
        self.average = self.average / columns
Let us examine some lines of the script.
  1. This line defines a class called TableAverageTask. A class is basically a bundle of variables and functions, called methods, acting on these variables. For more information on classes in Jython, see the Scripting Guide.
  2. This line sets the description of the task. It is what you see in the tooltip that appears when you hover on the task name in the Tasks view of HIPE. Every task must have a description.
  3. This line defines a task parameter. This parameter is called table, is of type TableDataset, is an input parameter and is mandatory.
  4. This line defines the description of the table parameter. Task parameters, like the task itself, must always have a description.
  5. This line defines the output parameter, called average. It is better to avoid vague parameter names such as result.
  6. This line defines the execute method. This is the heart of any task: it is the function that does the actual data processing.
Your task is now ready. You can now run it directly in HIPE or create a HIPE plug-in. Read the next two sections to find out more.

Saving the task to file

Follow these steps to save the task source code to a script file.
Make sure you are in the Workbench perspective of HIPE. If unsure, choose Window -> Show Perspectives -> Workbench perspective.
  1. In HIPE, choose File -> New -> Jython Script. A new window opens inside the Editor view.
  2. Paste the task source code in the new window. Make sure line indentations are preserved and no lines are broken. You may have to fix the code manually, especially if copying and pasting from a PDF file.
  3. Choose File -> Save as. Choose a file name with a .py extension and click Save.
The task is now saved to file. Read on to learn how to run it.

Running the task

Follow these steps to run your newly written Jython task.
  1. If your task is not open in HIPE, open it by choosing File -> Open File.
  2. Run the whole script by clicking the double arrow icon in the HIPE toolbar, or by choosing Run -> Run all. You should see no result apart from the whole script being echoed to the Console view. HIPE has read the definition of your task.
  3. Create an instance of the task by running this command in the Console view:
    tableAverage = TableAverageTask()
You can choose other names than tableAverage, but it is good practice to name the instance by dropping Task from the task name and turning the initial letter to lower case.
  • You can also add this line at the end of the script, without indentation, so it is automatically executed every time you run the script.
  1. Issue these commands to create a valid table dataset with three columns:
    x = Double1d.range(5)
    table = TableDataset(description = "A test table")
    table["First"] = Column(x)
    table["Second"] = Column(x + 1)
    table["Third"] = Column(x + 2)
  2. Execute the task by running this command in the Console view:
    average = tableAverage(table)
HIPE runs the task with table as input and puts the result in the new average variable.
For more information on running tasks from the command line, see the Scripting Guide.
If you close and restart HIPE, the task will not be available anymore. You will have to load and run the script file again. By turning the task into a plug-in, you can have the task available in the Tasks view each time you start HIPE. Read on to find out how.

Turning the task into a HIPE plug-in

Now that your task is complete, you can turn it into a HIPE plug-in, so that you and your colleagues can easily install and uninstall it from HIPE. To turn the tableAverage task into a plug-in, follow these steps:
  1. Copy all the source code into a file called plugin.py.
  2. Add the following lines to the plugin.py file. Note that these lines are not indented and not part of the TableAverageTask class:
    tableAverage = TableAverageTask() # 1
    toolRegistry = TaskToolRegistry.getInstance()
    toolRegistry.register(tableAverage, [Category.GENERAL, Category.SPECTRUM]) # 2
    del(toolRegistry)
    1. This line instantiates the task. That is, it creates an object from the TableAverageTask class. Note that from the task name TableAverageTask we obtained tableAverage as instance name. This is a general rule. The task name is made of words with capitalised initials and ends with Task. The instance name starts with a lower case letter and drops the Task part.
    2. This line registers the task instance, so that HIPE is aware of the task and can put it in the right places in the Tasks view. Here we register the task in the GENERAL and SPECTRUM categories. This means that the task will appear in the General and Spectrum folders in the Tasks view, in addition to the All folder. Of course this task has nothing to do with spectra: we include it in the SPECTRUM category for illustration purposes.
  3. Add the plugin.py file to a zip file.The zip file name must be made by three parts:
    • The plug-in name, for example MyAverage.
    • An underscore character.
    • The plug-in version number, which can be any combination of digits separated by dots. For example, 0.1, 1.0.0, 2.3.3 and so on.
The plug-in is now ready. Read on to find out how to install it.

Installing the HIPE plug-in

  1. In HIPE, choose Tools > Plug-ins The Plug-ins dialogue window opens.
    PluginsWindow.png
  2. Click Install new. The Install new plug-in dialogue window opens.
    InstallNewPlugin.png
  3. Click the folder icon and navigate to the zip file you have created. Click Open.
  4. Click Install. HIPE should inform you that the plug-in has been installed correctly.
  5. Restart HIPE.
The plug-in should now be installed correctly. Go to the Tasks view, open the All folder and type the first letters of the task name to jump to the right place in the list. You should see your new task. The task should also appear in any category you added it to, inside the By Category folder.

Executing the plug-in

  1. Double click on the tableAverage entry in the Tasks view. A dialogue box opens in the Editor view. Although we did not write a single line related to graphical interfaces in our TableAverageTask class, the task framework made one available for free.
    TableAverageTaskGui.png
    If you hover the mouse pointer on the table entry in the Inputs panel, you will see a tooltip informing you that the parameter is mandatory and must be a TableDataset. You will also see the short description we added in the source code. The same happens with the result parameter in the Outputs panel.TableAverageTaskTooltip.png
  2. Create a variable by issuing the following command in the Console view: notValid = Double1d([3.0, 5.0, 8.2]) This is not a good input for the task. Try dragging the notValid entry in the Variables view to the small grey circle next to the table entry in the task dialogue window. This is how you can set parameters via point and click, but in this case the small circle becomes red. In the source code we specified that the input must be a table dataset, so HIPE is rejecting everything else. We could have specified even more stringent conditions (for example, that the table dataset must not be empty).
    TableAverageTaskInvalidParam.png
  3. Issue these commands to create a valid table dataset with three columns:
    x = Double1d.range(5)
    table = TableDataset(description = "A test table")
    table["First"] = Column(x)
    table["Second"] = Column(x + 1)
    table["Third"] = Column(x + 2)
  4. Drag the table variable from the Variables view to the circle next to the table entry in the task dialogue window. The circle becomes green.
    TableAverageTaskValidParam.png
  5. Click Accept to execute the task.Two things happen:
    • The new variable average appears in the Variables view. This is the output of the task. Double click on it to inspect the output and make sure that the task did the correct calculations.
    • A new command appears in the Console view: average = tableAverage(table=table). This is the command line equivalent of executing the task from the graphical interface.
You have now successfully added your task to HIPE as a plug-in and made sure that it works correctly. The next step is to share your task with other users.

Sharing your task

You may want to share your task with your colleagues or with a wider user base. You have the following options:
  • To share your task among a small circle of colleagues: You can simply circulate by email the .py file with the modified task or the .zip plug-in file, if you created one.
  • To share your task among a wider circle of users: You can post the .py file with the task or the .zip plug-in file to the HIPE Community wiki. Open a free account, then click Pages and Files in the top left corner of the page to upload your file. Then edit one of the pages, or add a discussion post, with a link to the file you uploaded.
    AddFilesToWiki.png
  • To add your task to the official list of Herschel user-contributed software: Open a Helpdesk ticket with your request. For more information read the Herschel Data Products and Tools Contributor's Guide.
  • To add your plug-in to the official list of HIPE plug-ins: Send an email to Alvar GarcĂ­a.