Dear All, This is the summary of Terapix activities over the period from March 27 to April 7, 2009. The meeting will start today at 14:30, in the Salle des Seminaires room. Please let me know if points are missing, wrong or need clarifications. Amicalement, Yannick {{I. T0006, new images: Yannick}} ------------------------------------------------------------------------- Yannick transferred 850 images from CFHT that have been re-processed with ELIXIR using new flats. These images are part of Deep data and will have to be ingested, QFITS-ed and graded in Youpi. In parrallel, Yannick started to transfer all new Wide T06 images that were monitored and evaluated by Terapix over 2008 and split on all mix* nodes to fcix5. These images will then be put together with past (T0005) Wide images to start the T0006 release in Youpi (QFITS-in, Scamp, Swarp, Catalog production, QFITS-out and post-processing). {{II. Processing MegaCam: Yannick, Mathias S.}} ------------------------------------------------------------------------- No processing. Durell and NGVS still pending. {{III. Processing WIRCam : Patrick, Henry}} ------------------------------------------------------------------------- Henry: -- WIRDS: Henry focussed on the WIRDS data. Henry and Richd (Bielby) made considerable progess towards preparing the release pages for WIRDS. We did have a deadline of tomorrow but it depends on the CADC. Patrick: -- Laesker : Still no sign of download of the data !! Jerome Bouvier called Patrick last friday to discuss the various processing. He also wanted to know if old processes were still on disks to get some different stacks. We might emphasize that we keep the intermediate product only a limited period of time (one year). - Bouvier 07BF99 : Mix10 has been put back online. Not data lost on Patrick's side. All the stacks have been made. The final quality assessement is long because there 4 fields and 5 filters. Patrick expects to be able to release it this week. Patrick have started to write the report, which will be quite long, considering the number of fields/filters. BTW, the photometry is quite exceptionnal (around 0.01 shift compared to 2MASS). - Bouvier 08BF98 : There has been a missunderstanding between Loic and Jerome which ended with Jerome only sending me half of the data of this run. Patrick released the final products and Jerome noticed that some filters were missing. Loic put the rest of the run online and Patrick downloaded it. Some Scamps have to be redone because the data missing are on the same pointing and Patrick need a global solution including both the old and the new data. Up to now the new data are on the cluster disks (around 70GB) but Patrick has not yet started the processing. -- Lodieu : Patrickforgot to mask the region where no sky was computed before stacking the data. The new stacks will be produced today. The final photometry checks have been successful. The report is almost finished and Patrick expect to be able to release the run by the end of the week. -- Dubus : Two new requests from Guillaume Dubus have arrived. The PI is very friendly and sent the proposal with it. The two runs are each devided in two parts. Out of these four parts, one was not on the CFHT disks anymore but the PI will bring it to Patrick next week (he'll be in Paris). The rest of the data is on the cluster disks. Patrick has not started it yet. {{IV. YOUPI Tests/Optimis./Validation: Yulinana, Mathias M., Greg, Yannick}} ------------------------------------------------------------------------- Defining Youpi TOOO6-Wide input: *p.fits images, image lists (including pre-survey and photometric exposures), photometric flags, pre-Youpi qualityFITS output and grades (done by Yannick already), config files for QFITS-in, SCAMP, SWarp (TBD: SExtractor, QFITS-out), input directory structure. - Image lists - science, pre-survey, photometric: (1) all `*p.fits' images (2) split by field W1-W4 (3) new for T0006 (in progress) NOTE: The last image in T0006-Wide should be `1038843p.fits'. - QalityFITS-in: (1) list of all images, incl. pre-survey and photometric (2) path to flat-field images (3) path to bad-pixel masks (4) configuration file NOTE: The configuration file must contain `-SATUR_LEVEL 40000'. - SCAMP: (1) list of all `*.ldac' catalogues, incl. pre-survey and photometric (2) all `*.ahead' files (in progress) (3) configuration file NOTE: The configuration file must contain `-AHEADER_GLOBAL megacam.ahead' and the `ASTROREF_CATALOG 2MASS' parameters. - SWarp: (1) list of images per stack, i.e. per field per filter (2) configuration files per stack NOTE: The configuration files must define field center, as well as output image, weight-map and xml-file names according to the convention: `CFHTLS_W___TOOO6_*'. Testing Youpi performance and suggesting improvements (results will be reported by Mathias). FAQ: - Can we have output image lists from Youpi processing selections? TBD: - Store all `*p.fits' files in a single directory on fcix5. - Cross-check pre-Youpi QFITS image list with the Youpi input lists. - Cross-check if the grades provided by Yannick are included correctly in Youpi. - discuss and agree on the input data and data structure (with Yannick, Mathias and Greg) before starting the actual T0006 Youpi processing. - Set up an Youpi account for Mathias S. (@IAP: 13-17 April) {{V. YOUPI development: Mathias M., Greg.}} ------------------------------------------------------------------------- iSeveral points - mostly about improving Youpi in such a way it can better fit user's needs - have been discussed with Yannick and Yulia last week: Tasks implemented: - The image selector has been improved by adding a criteria reminder next to every line part of the query form. This way, one can never forget which criteria have been selected to build the current image selection (even if some widgets loose focus on the screen) - The default configuration files used by plugins (when relevant) are now the output of the -dd (was -d) option. - The results page have been improved: the search does not start automatically anymore. The user has to select filtering parameters first then hit the 'Start searching' button. Plugins can now merge their own filtering/searching criteria into the search form. This way, the qualityfits-in plugin adds a new filtering option based on image grading. All results related to QualityFitsIn now come with grading information, when available. Moreover, some stats showing the percentage of already graded images are available. Finally, any previously clicked row now remains highlighted (with a different color) so that remaining unsee/unchecked results can be retrived easily. http://clix.iap.fr:8000/youpi/results/ Tasks currently being worked on: - Add .ahead file support to Scamp plugin - Image selector: tag the image selection on-the-fly when uploading an image list. The new tag will be based on the uploaded file's prefix - Config file editor: allow per plugin config files ingestion . It will be possible to supply a data path to config files that will be ingested automatically. - Swarp: Input data validation bug - Results page: allow processing deletion - Redesign the statistics page - Output filenames automatic renaming (based on a concatenation of all common tags used for an image selection) - Image selector: add a new 'grade' search criterium (multi-selection) {{VI. Developpement processing: Patrick}} ------------------------------------------------------------------------- -- Remanence : On Lodieu data, one could notice a very strong remanence on the data. The saturated pixels on one image produced quite strong positive ghosts on the subsequent images (up to 5-10mn). The effect was so strong that one could see it easily even on the median stacks. I wrote a script that identifies the bright objects (2MASS online catalog) and do the masking of the next images. It still needs some tuning (how many images to mask) but the result is good. {{VII. Hardware/system/cluster Fred:}} ------------------------------------------------------------------------- {{VIII. Divers (All):}} -------------------------------------------------------------------------