Minutes of the March 14, 2007 VideoConf

Present: H. Aussel (OG), S. Baudet (CADC), J.-C. Cuillandre (CFHT), JJ Kavelaars (CADC), F. Magnard (Terapix), Y. Mellier (Terapix), C. Pritchet (OG), D. Schade (CADC)

-Part I: Ground Rules for the Oversight Group

We agreed on the following:

 1) The OG does NOT need to attend all videocons, and other meetings, but OG members may attend routine meetings as they choose.

 2) Either TERAPIX or CADC should have the option of asking (insisting) that they attend a meeting.

 3) A complete record of issues and actions items must be maintained and be available to all parties.

This means any emails, telephone calls, face-to-face meetings, videcons etc, need to be summarized (briefly) and action items and agreements need to be written down and put in the archive.

If an agreement or action is not written down and accessible in this archive, then it is NOT an agreement or action. It has no effect.

 4) Either party (CADC or Terapix) may declare an impasse or other difficulty that requires the attention of the OG. In this case CADC , CFHT or Terapix are free to appeal the OG. Then

 - 4.a) The OG will examine the situation, and propose a solution by consensus.

 - 4.b) If the OG cannot reach conscensus in the matter at hand, the party that required the OG attention should refer to its Agency.

 - 4.c) If the solution proposed by the OG is not satisfying, the parties should appeal their agencies.

 5) The OG will have access to the archive of communications and committments

 6) CADC and TERAPIX agree that the decision of the OG is binding and that they must act in accordance with that decision.

The ONLY alternatives are:

 - a) appeal to the Agencies

 - b) informing all parties that they intend to withdraw from their commitments under the MOU between CADC, CFHT, and TERAPIX

CFHT will be responsible for setting up a system to archive and make accessible the list of action items and decisions.

- Action items:

 In order to keep the one-to-one interactions between Fred, JJ and Jean-Charles, JC will re-activate the Data Operation Group (DOG) mailing list

 JC will set up the system at CFHT to archive action items and decisions. It should be simple: minutes will be included with a set of actions open/closed. The CADC/CFHT/Terapix participants to videoconferences (in particular the the DOG), the OG. The SG will have access to the archive of the minutes.

Part II: Technical issues

The following summarizes the first issues/actions raised during the second part of the video-conf.

- I. Missing files:

Issue:

There were a number of files that were being hidden from the megaprime_proxy list. These images are parts of 'incomplete' datasets where we have the CAL and RAW file but complete set of QFits output was never received. Because parts of the complete dataset were missing the proxy was skipping the files as incomplete sets.

Action from JJ: I've changed the logic so that now the file is offered regardless of completeness... for some of the datasets this will cause a problem as the part of the dataset that is missing is the ELIXIR processed image from CFHT.

- II. RICE compression:

Issue: The MegaPrime_proxy's output contains a URL which pointed towards a gzip compressed version of the image (efficient compression ratio, but 10x slower to uncompress, and invisible headers).

Action from JJ: the URL that is provided as part of the proxy has been changed to deliver .fz files instead of .gz files. This should improve your speeds a bit.

- III. Skyprobe:

Issue : JJ report on the frequent missing skyprobe information, though it is used by Terapix. This is because the CFHT FITS table is sometime FITS compliant , sometime not.

Action for CFHT: CFHT will investigate why it not always FITS compliant and will make the FITS table stable.

- IV. CADC-Terapix data transfer protocol:

Issue: Terapix-CADC data transfer is done through CADC's e-transfer method (using bbftp), which uses "new" directory for files never transfered (triggers an error if a file placed in this directory already exists @cadc). And a "replace" directory for files to replace (triggers an error if the file is new). Terapix asks to use a single directory such that existing file is replaced, and new files are downloaded without triggering errors. Terapix also asks to have one e-transfer daemon running per machine (5 machines dedicated to the pipeline at this time: mix3, mix4, mix5, mix6, mix8).

Action from Terapix and CADC:

Terapix needs to open to the outside world the necessary ports for bbftp to work on the 5 machines. CADC needs to set up e-transfer on the 5 machines. Terapix and CADC have to agree on which directory to use.

CADC will install a more recent release of e-transfer onto CLIX.IAP.FR as the version currently on that machine is out of date. A timeline for this is TBD, but it should be on a short timescale.

- V. Terapix to CADC STEP-1 data transfer:

Issue: need to clarify what terapix should send to CADC and in which form

Actions: Terapix will provide individual FITS Weightmaps images (FITS compliant) and a dump of the Terapix DB that includes the Grade and the Comment for each image in a CSV format when the release is frozen. This transfer will be done through the already set-up e-transfer protocol.

- VI. Need for Checksum attached to each image:

Issue: Terapix found several image that was corrupted after the transfer from CFHT to Terapix. Fred proposes a checksum be created for each image as soon as it is created.

Action: CFHT, CADC and Terapix will agree on a checksum format for all CFHTLS and PI data that will be created by CFHT and Terapix, and included in the FITS header. Cf. http://heasarc.gsfc.nasa.gov/docs/heasarc/fits/checksum.html (which implementation do we choose ?) fitsverify can verify checksum consistency (cf. http://fits.gsfc.nasa.gov/ErrorList.html ) Each file's checksum will also be available to Terapix through the CADC's megaprime_proxy.

- VII. Next meetings and dead lines for actions:

The next meeting will be same time as the current one but 2 weeks later. That is March 28th at 10AM PDT. whenever possible this will be the deadline for the actions listed above.