Dear All, This is the summary of Terapix activities over the period Jan. 23, 2009 to Feb. 11, 2009. The meeting will start today at 14:30, room 281. Please let me know if points are missing, wrong or need clarifications. Amicalement, Yannick {{0. Specific Yuliana's training: Yuliana; with help from Patrick, Yannick}} Training MegaCam data processing (Hudson's data set - 08AC24-Hudson): - producing chi2 images with SWarp completed - Q.C. of the stacks: STIFF ugi-colour images; direct comparison with the Hudson's data release (downloaded from the ftp server) - object detection & photometry for QualityFITS_out (SExtractor in single-image mode) in all fields and filters - SExtractor runs with larger set of PHOT_APERTURES for growth curve analysis {{I. T0005 and T0006: Yannick, Mathais S.}} T05/T06: Mathias S. started to look at the overlapping regions of the W4 fields and to map it in E-W and N-S direction (see my plots). The offsets determined are very stable with different mag_lim. I still have to check how the offsets in the overlap region correlate with the offsets found in SDSS. {{II. Processing MegaCam Yannick, Patrick}} Vergani_08BF26: Yannick and Patrick have produced data products and documents for the MegaCam GRB data. Still some work to do, after a mis-use of keywords in the joined SCAMP step, but close to completion (see Patrick's WIRCam processing summary). Kong_08BT10: Yannick produced and delivered all output on ftpix. No feed back from the PI yet. Nothing special; rather easy processing. - NGVS: Yannick asked Jean-Charles Cuillandre for clarifications about the sky background sub-traction in next NGVS data. Jean-Charles answered a sky model built by CFHT is planned, so in principle, it will ease the data processing Terapix. However, Jean-Charles did not anwser about the way the sky background images have to be handled together with weight map images. Jean-Charles also asked Yannick to do their own sky subtraction anyway to be sure Jean-Charles recipes work. Jean-Charles will send details soon. {{III. Processing WIRCam : Patrick, Yuliana}} Yuliana started WIRCam data processing: Ibata's data set - IBATA_08BF13): - start-up with WIRCam reduction procedures (with Patrick) - Q.C. of the sky-subtracted images: visual inspection of the images, QualityFITS seeing - QualityFITS problems (no plots produced) in ~3.5% of the images; Condor tests on each of the cluster mix* nodes, problems on mix2; redo QualityFITS for these images - all OK Patrick WIRCam processing activities: - Flaguey : The release is virtually ready. I was waiting for Henry's comments (English) but since Yannick already gave me some comments, Patrick will probably release the data/report today. - Vergani : The joined SCAMP was wrong. The ZP keyword is different in WIRCAM and MEGACAM. Therefore it has to be copied (and replaced by the other keyword) in a ahead file for each image. The consequence was an incorrect photometric rescaling of all MEGACAM images. The bug has been corrected and the new SCAMP has been calculated. Both MEGACAM and WIRCAM stacks are ready. Patrick will send today the stack to Yannick to build the Chi2 image (or do it himself). What remains to be done is the regular final quality control on the images. The PI also asked for completeness on both instruments data. On WIRCam images, the completeness varies quite a lot because of the dithering filling the gaps between the CCD. Patrick created the simulated images for the completeness calculation in NIR (J,H and Ks). - Barmby : Patrick has tested on this run (with very bright galaxies) a version of the sky substraction which allows the complete discarding of a chip because of a very bright object. The result is much better than by only masking part of the CCD. All objets (4 fields) for which the dithering pattern allows this method have been reprocessed. Regarding the skysubstraction, Patrick has applied the optimal method he has. A remaining issue is the crosstalk. CFHT first released a version were the XT removal step had been skipped. Then Patrick reprocessed it with the XT correction. The problem is that the simple procedure of the CFHT yields bad results when there are bright galaxies on the field. It leaves vertical pattern due to an incorrect XT background estimation. Patrick is trying to apply his own correction and check if he can do better on this particular point. The writing of the report is underway. {{IV. YOUPI development: Mathias M., Greg.}} Implementation of Swarp plugin is complete (but still needs real case testing): support for .head files has been added and total exposure time is now displayed on the results page: http://clix.iap.fr:8000/youpi/results/swarp/6079/ From the results page, the 'reprocessing' feature has been implemented for both Scamp and Swarp plugins. All shopping cart items resulting from a reprocessing action (from the results page) are now displaying a link to the configuration file used for the same processing. This can be very useful if one wants to be sure that an item is using the right configuration file prior to submit it to the cluster for processing. The 'release' concept is being added to Youpi. The database has been heavily modified to support it. The 'release' will be a major improvement to Youpi since it will allow users to organize their data. Without it, it would be impossible to use data from T006 only for example. There would be no way to distinguish subsets of data. In practice, a release is just a user-defined name (with an optional comment line) to which ingested images can be associated. From the new top-level 'Release' menu, one will be able to choose - and edit (by adding or removing more images) - a release to use throughout the YOUPI interface. Therefore, the active (or working) release (the one you just selected) will be used at every stage: when selecting images (plugin processing) or after a successul processing in order to bind the results to the current release (for example to know that this SWARP has been made on T006 data) etc. As of today, only a partial implementation of the 'release' feature is available. Tests are being made with the Swarp plugin. The image selector object has been enhanced to support releases. As a reminder, the current (working) release name is always displayed on the top-left corner of the interface, just below the login name. A short demo is available at http://clix.iap.fr:8000/youpi/processing/swarp/ Mathias M. thinks that about 2 more weeks of development are needed in order to have a full implementation (and integration) of the release feature. {{V. Developpement processing: Yuliana, Patrick, }} Yuliana software developement activity - Yuliana is developing a tool for growth curve analysis based on SExtractor PHOT_APERTURES & MAG_APER parameters; - some normalization problems at the beginning - Yuliana switched to the procedures described in Howell (1989) and Stetson (1990); - detailed tests on the Hudson's data set: growth curves for point -like sources, preselected from stellar locus in the MAG_AUTO vs. FLUX_RADIUS plots -> stable 'optimal' aperture over a wide range (16-22) of magnitudes; testing the same procedure for extended sources in different MAG_AUTO and FLUX_RADIUS bins -> dependence on both parameters, more strongly on the FLUX_RADIUS; no corrections have been implemented yet. Patrick software developement activity: - Skysubstraction : Debugging and testing of the skysubstraction with one chip discarding capabilities. This version could be useful for the NGVS processing. Ahead files handling : A Python library and some scripts to handle ahead and head files. Completeness : New simulations for NIR filters Script option to simulate only stars (useful when the image is too small to get a good statistic with the default low density of stars). It would be useful to have a tool that could plot a continuous 2D surface of completeness on an image. {{VI. Hardware/system/cluster Fred:}} {{VII. Divers (All):}} - Yannick got very positive comments from the French CNRS/INSU/CSAA about Terapix activities. CNRS/CSAA congratules the Terapix team as a whole for their work. For those who are not familiar with these acronyms: - CNRS: Centre National de la Recherche Scientifique - INSU: Institut National des Sciences de l'Univers (a CNRS national institute that operates all French national facilities, TGEs (Tres Grands Equipements, like CFHT) and organises all astronomy projects in France. - CSAA: Commission Specialisee Astronomie et Astrophysique. An Advisory Committee that advices INSU about all astronomy projects in France. - The PNCG (Programme National Cosmologie et Galaxies) Scientific Council met last week. No official outcome have been issued yet, but Yannick got an unofficial feed back: we should get 15keuros, as requested, for Terapix. We should also get 9 keuros for NGVS, as requested as well, but conditional: if the ANR proposal is rejected we will get 9 keuros from PNCG, if the the ANR is accepted we will get the 9 keuros from the ANR (this conditional support was asked by us). So, if everything is set as planned we have: 33 keuros from INSU/CSAA 15 keuros from PNGC 09 keuros from PNCG or ANR 05 keuros by IAP (but already used for the new IAP server) Note that we also have extra-funds from the ANR "DESIR" led by Henry. - On behalf of Terapix, Yannick sent an email to CFHT about our concerns regarding some observing strategy (see email below). No feed back yet, but C. Veillet conformed they are working on it. email to Veillet: Dear Christian I would like to raise an issue we often face on during the Terapix processing of PI programs, that very recently stuck the analysis two important CFHT projects. Over the last three weeks we processed the WIRCam 08AF15 (PI Flaguez) and the WIRCam (08BF25)+MegaCam (08BF26) (PI: Vergani) data. In the first case, the data processing turned out to be very challenging because the OFF field used for sky subtraction was as crowded as the science center field. Terapix spent several weeks to find an optimal solution in order to get the best possible sky background, but failed. The comparison of the WIRCam photometry with 2MASS clearly show it is not good and far from the Terapix standards. In the second case, the PI wanted to observe a GRB with WIRCam and Megacam. However, the PI centered the observations exactly at the GRB center position, which most likely corresponds to the CFHT WIRCam and MegaCam optical axis... that is within the gaps of the WIRCam and MegaCam detectors. Due to the small dithering between each exposure, Terapix marginally found a very faint host galaxy at the GRB position, but it is extremlly faint and not visible on all stacks. Getting reliable photometric information from these data is not possible. We believe the PI did not realise an offset of the telescope was necessary in order to avoid the gaps. Fore sure, it is up to the PI to carry out the definition of pointing for the QSO that suits its scientific needs. On the other hand, since service observing is a more and more common practice for new generation astronomers, they may miss this kind of technical issue and it would be most valuable we (CFHT and Terapix) anticipate them prior to start observations. In both cases, we could avoid these mistakes. Some warnings could be put in the proposal forms. It would be also most valuable if Terapix and CFHT have a joined technical review of each accepted proposals to check the feasibility. Terapix can help in particular for post-acquisition issues related to the data processing, even if at the end the PI does not request any help. This is additional works for us, but certainly less than the extra works needed to carry out hopeless and endless data processing at Terapix. We will also avoid bad surprises for the PIs , we will make them happier and we will avoid waste of telescope time. We would like to have the views of CFHT on this. Thank you very much in advance for your advices and comments. Best Regards Yannick, on behalf of the Terapix team