HIPE Advanced Usage telecon # 2

October 18, 2011


  • David Ardila (DA)
  • Jeroen de Jong (JdJ, partially)
  • Carolyn McCoey (CMC)
  • Yi Mei (YM)
  • Bruno Merin (BMe)
  • Stephan Ott (SO, part time)
  • Davide Rizzo (DR)
  • David Shupe (DS)

1.- Welcome and Agenda

Goals of the meeting are:
  • to review updates on 8 and to find bottlenecks that need work
  • more in general, to serve as communication channel between HSC development and users

The agenda is:

1.- Welcome and Agenda
2.- Updates on HIPE 8
2.1.- The plugins framework (video tutorial)
2.2.- Summary of available contributed software (from HSC web)
2.3.- Interoperability improvements in HIPE 8 (DS9, and more)
3.- Work being planned for future versions from DPUG 26
3.1.- MyHSA and data access
4.- User experience coding in HIPE
5.- Questions, comments

Any further contributions?

2.- Updates on HIPE 8

HIPE framework updates: What's new HIPE 8

2.0 Threads from previous meeting

CMC: Even for experienced users, advanced HIPE features [as presented by Jaime at the HIPE forum, PB] are unknown.
Follow-up: HCSS-13743 -- implemented in HIPE 8

MK-1: Basic tasks are obscure to users... It should be easy to get a list of observations in pools and storages and to know their versions.
The version issue is raised later. Getting a list of observations in a simple way is done in HIPE 8. This was demonstrated.

MK-2: There should be easy buttons to add and remove pools.
Follow-up: HCSS-14243 This will be planned for HIPE 9.

MK-3: Versions of observations are confusing. The most recent version is not always the highest number.
Follow-up: HCSS-13781 Not yet understood, under investigation.

MK-4: It would be nice to have numpy available.
Follow-up is still pending. SO mentions that he believes Juan Carlos and Jorgo have found problems with this due to differences in data structures between ia_numeric and numpy.
Action PB: Take this up with Juan Carlos and Jorgo.
DS: I'm a big user of numpy. What would be nice is to have a dictionary that references numpy functions to ia_numeric functionality.
DS: You can register Python as a SAMP client and exchange data with HIPE, but there are still some details that aren't clear. Send the info that is available now about how this is done, together with the open questions, to PB.
Post-meeting comment: Discussion started on Advanced usage of HIPE page (discussion tab).

JySTILTS (Topcat integration using Jython): Pending. No clear user demand but this may be due to lack of awareness of the possibility.

Presentation of constributed software is in different places: From community support and DP HIPE plug-ins
Follow-up: These pages are reference each other.

2.1.- The plugins framework (video tutorial)

Davide Rizzo has published an introduction on making simple plug-in on Youtube: Youtube video

An overview of the improvements in the HIPE Plug-in Framework in version 8.0 can be found in JIRA.

2.2.- Summary of available contributed software (from HSC web)

The available plug-ins that the HSC knows about can be found from HIPE via Tools --> Plug-ins --> Find more plug-ins.

2.3.- Interoperability improvements in HIPE 8 (DS9, and more)

It was already possible in HIPE 7 to send and receive data from DS9 but in HIPE 8 this will be enhanced with some preset information. This was demonstrated.

DA: What about multiple planes?
PB: SimpleImages normally have only an image plane so this is not normally an issue. But if you have an image with 3 separate planes for RGB, then DS9 will only show you the first one even though HIPE sends all 3.

3.- Work being planned for future versions from DPUG 26

We discussed several of the highest priority issues from the list of issue raised on the HIPE framework during DPUG 26.

First about variations in the Task syntax and the fact that some data structures allow Jython style access, whereas others don't have any Jython hooks and must be used Java-style.

BMe: I believe that the fact that sometimes you get the number of elements using x.length, sometimes using x.getLength(), sometimes using x.size() -- that this is an obstacle especially if we want our code to be useful for missions after Herschel as well.
CMC mentions that for data structures, such as Spectrum2d, HIFI users have become used to what is there and changing this would be painful.

So we move on to Task behaviour. For Tasks to have homogeneous behaviour, the desired behaviour should be defined in a style guide.

BMe: For Tasks, the issue is that sometimes the input parameter is being modified, sometimes a new variable "result" is created (and the next task creates result1, result2, result3 -- which complicates knowing what was what).
CMC: I agree that this can be confusing. The DPUG minutes said that this situation should be well documented but I have not yet seen any HIFI documentation SCRs on this issue.
DA: If homgenizing all existing tasks is too much effort, can we at least enforce this for new tasks?
DR: We have defined the desired behaviour but we need enforcement.
PB: Javier Diaz has raised SPRs on Tasks not following conventions for parameters names (i.e. parameters not being lower-camel-case).
DS: I've not considered varying Task behaviour as a serious problem so far but I have been struggling with parameter names.

3.1.- MyHSA and data access (tbc)

  • HSA searching from the Product Browser
  • HSA cached data from Preferences

Default look of the Product Browser (in process of being integrated):
  • All local pools selected
  • HSA cached data selected
  • Hitting "Run" will just show all observations on local disk

After selecting HSA on-line you can query the actual HSA. However, this will return a very large result set, so you will hit a protection:


CMC: This error is really high-tech and not very appropriate to show to users. We should just show the second line "Your query exceeds...".
BMe: Agreed.
Post-meeting comment: SCR raised: HCSS-14370

Instead, make a selection, e.g. by specifying you want the observations from OD 200:


Further plans:
  • For 8: getObservation upgrade
    • Read from any data source (HSA across the web, HSA tar, zipped pool, ...)
    • My HSA integration
  • For 9: Product Browser as "iTunes" for Products:
    • Complete library management.
    • Create, remove and rename pools
    • Drag and drop of products from search results to copy/move products between pools. (Removal already exists, including recursive removal.)

YM: What about saving data? Will we get one single command for exporting to different formats as well?
BMe: The single command for reading will be getObservation, the single command for writing/exporting will be saveObservation. It is not clear how much of this will be available in 8.0 and how much will have to be postponed to 9.0, but this is the plan.

YM: We cannot delete pools from the Preferences --> Data Access --> Storages and Pools panel unless the pools are empty.
PB: This was consciously put in to protect users.Clearly deleting a pool that is not empty can potentially delete lots of data. We want to protect the adventurous users that try new buttons to see what they do, from disaster.
BMe: That makes sense in principle.
YM: This is inconsistent with the fact that we do allow users to delete observations and everything underneath from the Product Browser though. If we allow user to delete data from the Product Browser, it should be possible to delete an entire pool, rather than forcing the user to delete all observations in the pool one by one (plus that deleting all observations does not guarantee that the pool will be empty...).
PB: If we allow this we should add a warning along the lines of "data will be removed -- this cannot be undone".
All agree that this protection should be removed: Deleting pools that contain data should be allowed, but a warning should be shown.
Post-meeting comment: SCR raised: HCSS-14371

PB: It should be possible from the Product Browser to create, remove and rename pools. The Product Browser should be for your pools and products like iTunes for your media: A complete "library manager".
CMC: In the same analogy, can the browse image serve as "cover art"?
PB: Nice idea, it probably can.
Post-meeting comment: SCR raised: HCSS-14372

4.- User experience coding in HIPE

I have just add a new section to the Plug-in Developer's Manual on how to create a plug-in with a Task. Any feedback on that already?

Any feedback in general? Positive and/or negative experiences?

From Davide Rizzo: See also these other tutorials:
My intention is to add more tutorials like these and improve the existing pages in the Contributing to HIPE section of the official Herschel wiki.
I am considering moving these pages to this wiki, to turn them into a real community effort. For this documentation to be really useful, I need the community to take an active role by doing the following:
  • Commenting, pointing out errors, requesting new tutorials.
  • Editing pages (if a page is open to editing, you do not need to ask for permission).
  • Contributing examples, tips, workarounds and so on (you will be duly acknowledged).
I would like to use the permanent beta concept for this documentation: get out something usable now, rather than something perfect six months later. For this to succeed, community involvement is essential.

DS: I have a technical question about plug-ins. It seems that plug-ins are geared towards easy installation, but I find development hard. It's not easy to debug a plug-in.
PB: I do this as follows: From the Plug-ins panel (that you open from the Tools menu in HIPE), you select the plug-in that you want to debug. From the View menu on the Plug-in panel, you can select "Browse plug-in". This opens a file chooser that points into your installed plug-in. From there you can open any deployed plug-in script. It will open in the editor where you can edit it, and run it to redefine the objects that your plug-in provides to HIPE (such as a Task). You can then quickly iterate by trying the task, making more edits, and so on.
If more info is desired about this, please start a discussion on the Advanced usage of HIPE page ("discussion" tab).

5.- Questions, comments

BMe: There have been reports that Jython 2.5 breaks the import statements in your Jython scripts. Users claim that they can't upgrade to HIPE 8 because of this.
DR: This is documented on the Jython upgrade Wiki

6.- Telecon info

Meeting information
Topic: Advance HIPE usage
Date: Tuesday, October 18, 2011
Time: 5:00 pm, Europe Summer Time (Amsterdam, GMT+02:00)
Meeting Number: 848 917 327
Meeting Password: 123hipe456

To start or join the online meeting
Go to https://esa-meeting.webex.com/esa-meeting/j.php?ED=183827012&UID=491183002&PW=NOWQ2NGI5ZDg4&RT=MiMyMg%3D%3D

Audio conference information
To receive a call back, provide your phone number when you join the meeting, or call the number below and enter the access code.
Call-in toll-free number (UK): 0800-051-3810
Call-in toll number (UK): +44-20-310-64804
Global call-in numbers: https://esa-meeting.webex.com/esa-meeting/globalcallin.php?serviceType=MC&ED=183827012&tollFree=1
Toll-free dialing restrictions: http://www.webex.com/pdf/tollfree_restrictions.pdf

Access code:848 917 327

For assistance
1. Go to https://esa-meeting.webex.com/esa-meeting/mc
2. On the left navigation bar, click "Support".
To add this meeting to your calendar program (for example Microsoft Outlook), click this link:

To check whether you have the appropriate players installed for UCF (Universal Communications Format) rich media files, go to https://esa-meeting.webex.com/esa-meeting/systemdiagnosis.php