Dear All, This is the summary of Terapix activities over the period from May 04 to 13, 2009. The meeting will start today at 14:30, room 281. Please let me know if points are missing, wrong or need clarifications. Amicalement, Yannick {I. T0006 : CFHTLS-Wide T0006 data management & Youpi I/O setup: Yuliana, Mathias M., Yannick}} ------------------------------------------------------------------------- Cross-checks of image lists, available data and data ingested/QFITSed in Youpi is finished now. The summary tables bellow are split by field (W1, W2, W3,W4), data type (SciW, PreW, Phot) and data release (T0005, T0006). Please note that 'W0' denotes data in all Wide fields and 'Full' refers to all types of data. ----------------------------------------------------------------------- - Full T0006-Wide *p.fits data set (6045 images): W0 W1 W2 W3 W4 T0005 T0006 SciW 4849 2059 698 1352 740 4040 809 PreW 855 341 145 242 127 633 222 Phot 341 176 26 100 39 268 73 ---- Full 6045 517 171 342 166 4941 1104 ----------------------------------------------------------------------- - *p.fits files missing on fcix5 (669 files): W0 W1 W2 W3 W4 T0005 T0006 SciW 1 0 0 1 0 1 0 PreW 588 277 98 213 0 588 0 Phot 80 40 2 29 9 7 73 ---- Full 669 317 100 242 9 596 73 ----------------------------------------------------------------------- - *p.fits files not ingested yet in Youpi (1745 files): W0 W1 W2 W3 W4 T0005 T0006 SciW 810 452 16 239 103 1 809 PreW 855 341 145 242 127 633 222 Phot 80 40 2 29 9 7 73 ---- Full 1745 381 147 271 136 641 1104 ----------------------------------------------------------------------- - *p.fits files not QFITSed yet in Youpi (2401 files): W0 W1 W2 W3 W4 T0005 T0006 SciW 1437 452 17 240 728 628 809 PreW 855 341 145 242 127 633 222 Phot 109 40 2 29 38 36 73 ---- Full 2401 381 147 271 165 1297 1104 ----------------------------------------------------------------------- - grades (on-going): Multiple entries found in the new T0006 grade list `T06-New-Grade.csv' for 92 images. Clarified with Yannick and cleaned. In addition, cross-check with the Spica grades (provided by Mathias M.) shows further double entries for 29 images. Most probable reason (after discussion with Yannick and Jean-Charles): new grades by Yannick of images which have been re-processed at the CFHT due to flatfielding problems. The new grades have been kept. All corresponding csv lists (containing odometer number, object, field, filter, data type, data release) are available at: /home/nis/goranova/work/DataReduction/Youpi/CFHTLS-Wide/T0006-Wide/ImagesList/ CFHTLS-T0006-Wide-VALIDATED-Full-W0.csv CFHTLS-T0006-Wide-MISSING-pfits-Full-W0.csv CFHTLS-T0006-Wide-MISSING-youpi-Full-W0.csv CFHTLS-T0006-Wide-MISSING-qfits-Full-W0.csv CFHTLS-T0006-Wide-GRADES-Check-T0006.csv CFHTLS-T0006-Wide-GRADES-Check-T0005.csv {{II. Processing MegaCam: Yannick, Mathias S.}} ------------------------------------------------------------------------- {{III. Processing WIRCam : Patrick, Henry}} ------------------------------------------------------------------------- WIRDS (Henry): - Thanks to the help of Jean-Charles, the origin of the rectangular- shaped underdensity in the photometric redshifts between 1 fixed - in ingestion and preingestion step -> in progress Fixing minor bugs in configfilewidget display Mathias-led: User permissions implementation into Youpi is on its way. We largely discussed the point on the teralist in the last few days. Many Thanks to all Terapixians for their useful comments! User/Group permissions is a rather big part to implement inside Youpi and we know it is blocking for Yulia and Mathias S. (as they cannot work together reliably for now because Youpi does not allow user data sharing). Several parts of the Merise Analysis have been added to support groups and permissions. The new conceptual model of data (MCD) is available at http://terabsd.iap.fr:22080/youpidoc/handbook/ch04s02.html As you can see on the picture, the new red part of the graph is related to user/group permissions. Those changes implied modifications in the following entities: Cart Item, Image Selections, Config Files, Misc Data, Config Type, Ingestions and Processing Tasks. That means that these entities will support UNIX-like access permissions. Each of them will be assigned a user (or owner), a group and an octal access mode. For example, I can set my Scamp processings to be chmoded 640 with the group 'Terapix' which means that I can read/write them, that the 'Terapix' group can read all of them and that no permission is granted for the rest of the world (others). This way we will be able for example to make a release public by simply chmoding some processing results. As for UNIX-like permissions, only the owner can change permissions for its data and only an administrator (memmber of the 'admin' group) will be able to change some data's owner (chown). For now, Mathias M. is working on implementing permissions for the tags. It is mostly functionnal at http://youpix.iap.fr:8000/youpi/tags/. Since the Tag entity is the first one to support permissions, it took quite some time to implement, but core permissions API is now set so we will be able to add permissions support to the remaining entities more easily. Finally, Mathias M. would like to know whether - anywhere it is applicable - warnings should be displayed on the page when some data is filtered (not visible) so that one can know that only some parts of the information is available to him. {{VI. Developpement processing: Patrick}} ------------------------------------------------------------------------- - Patrick has written a new version of the parallelized skysubstraction which allows the user to provide both list of images (to be skysubstracted and to compute the sky). - The very simple skysubstraction procedure (images medianing) has to be improved. A first improvement would be to allow a scaling of the sky images (according to the background sky leven) before the combination (as it is done at CFHT). A substraction of large scale gradients before the combining could also be activated on NOT crowded fields where extended object should not contaminate the sky. - Patrick read further doc on Flex/Bison. Both notations (Inverse Polish and regular ones) can be implemented quite easily. - Patrick is reading the source code of Emmanuel's SPRED (which uses Flex/ Bison) and his fits libraries which we will probably use to speed up the handling of the fits images. {{VII. Hardware/system/cluster Fred:}} ------------------------------------------------------------------------- {{VIII. Divers (All):}} ------------------------------------------------------------------------- - Tests were made by Henry with Emmanuel's new versions of Sextractor and PSFex. Henry noticed that these programs were not up-to-date on the cluster. It was not possible to compile PSFex on the cluster because the version of PLplot seems to be old. In the end, Henry compiled PSFex on his mac (the new version of PSFex provides several very useful check-plots which the installed version does not). Bulge+disk galaxy models were fitted on a ACS model tile with a view to comparing this to the current COSMOS morphological catalogue. - Emmanuel provided an updated sextractor binary which should work without problem on mix-x where x>5 ; the older mix machines do not have the required SSE instructions, so it seems that a special compile would be needed. Henry also mention that Emmanuel has indicated on the TERAPIX web pages the new home for software packages. - Yannick and Henry often see php error messages on the TERAPIX web pages.