Dear All, This is the summary of Terapix activities over the period from May 14 to 26, 2009. The meeting will start today at 14:30, room 281. Please let me know if points are missing, wrong or need clarifications. Amicalement, Yannick {{Ia. T0006 : CFHTLS-Wide T0006 data management & Youpi I/O setup: Yuliana, Mathias M., Yannick}} ------------------------------------------------------------------------- - All T0006-Wide *p.fits file copied to fcix5/raid1. Data retrieved from CADC archive were fits-verified. Check-sums computed for all images. Data distributed in sub-directories per field: /data/fcix5/raid1/youpi-INPUT/CFHTLS/T0006-Wide/Images/W1 /data/fcix5/raid1/youpi-INPUT/CFHTLS/T0006-Wide/Images/W2 /data/fcix5/raid1/youpi-INPUT/CFHTLS/T0006-Wide/Images/W3 /data/fcix5/raid1/youpi-INPUT/CFHTLS/T0006-Wide/Images/W4 - Input-data statistics: field number of images disk space W1 2576 1.7T W2 869 573G W3 1692 1.1T W4 906 598G Total 6043 3.9T - Following (validated) images were not found in the CADC archive, to be excluded from the reduction: 973747p.fits W3, r, Phot, T0006 987371p.fits W3, i, Phot, T0006 {{Ib. T0006 : CFHTLS-Wide T0006 Youpi processing: (with Yannick, Mathias M., Greg, Fred, Mathias S.): ------------------------------------------------------------------------- - Some problems with Condor and fcix5 - solved. - Pre-ingestion: problems with mcmd*.fits tables for the new T0006 images - Ingestion: so far successful for the old T0005 images (on-going) - Youpi-OUTPUT currently being moved from mix21 -> fcix5/raid2 (to be reported by Mathias M.) Mathias S. is updating his scripts for post-processing of T0006: merged catalogs, colour-colour plots, completeness plots and comparison with SDSS Major changes: For the colour-colour plots the basel 3.1 stellar library will be used as reference (and not "Pickles" as before) {{II. Processing MegaCam: Yannick, Mathias S.}} ------------------------------------------------------------------------- {{III. Processing WIRCam : Patrick, Henry}} ------------------------------------------------------------------------- WIRDS (Henry): - With Patrick's help the COSMOS-H runs 08BH52,08AC99,07BC99 have been processed to sky-subtracted images - This data will be added to the 08AF97 and 08AH24 runs processed in November - In total there are 10,000 images. This represents the entirety of the COSMOS-H band data. - The objective is to have a final cosmos-H stack for the 4th of June (for the COSMOS meeting in Japan) - Loic has been contacted to provide reprocessed 05 and 06 COSMOS data but apparently this will require a lot of work (perhaps for the end of the summer it will be ready?) - Rich continues to process the D3 data. - No news from Loic about the Azimulu data photometric data. - D1 photometric redshifts computed with the new catalogues (using T0003 r,z frames) do not show the same detector anomaly present on the previous version of the photo-zed catalogue. - note: Patrick's scripts do not function correctly on the new nodes. Is this a problem for Fred or a problem for Patrick? WIRCam (Patrick): # WUDS : - J and Y images have been skysubstracted. I am checking the results. - Ks skysubstraction is running now. # Dubus : - All pointings reduced and stacked. The PI asked for stacks for individual lists in addition to global ones (one stack per list and not one stack per pointing). They still have to be done (and the corresponding qualityFITS). The results then consist in more than 50 stacks (3 bands and 9 pointings) !! - The report has to be written. # Soucail : - All images have been skysubstracted. Half of the Qfits have been made, the rest is running. I am checking the results. {{IV. YOUPI Tests/Optimis./Validation: Yulinana, Mathias M., Greg, Yannick}} ------------------------------------------------------------------------- See Section Ia of this report. {{V. YOUPI development: Mathias M., Greg.}} ------------------------------------------------------------------------- Greg-led: Youpi: SExtractor bugs: - Fixing issues for displaying modalbox (3rd party js code) - Better handling single/dual mode (display problem) - Adding a dropdownbox in order to display the parameters file (same as config file) http://youpix.iap.fr:8888/youpi/results/ -> sextractor General bugs: - The ingestion was broken since the structure of the condor requirements has changed -> fixed - For the production database : drop the youpi_fitstables do preingestion by hand, due to the need of T0006 release -> using scripts, which read Yannick's big log files and refill the youpi_fitstables table. Improvements: Join Mathias M. + Greg Work to automatically ingest configuration files (for all plugins ) contained somewhere in the cluster, by giving a path through file browser js class. Mathias-led: User permissions implementation within Youpi is over. Groups of people can now work together and share data. Unix-like permissions ensure a good level of privacy. As of today, permissions are available for tags, shopping cart items, saved image selections, saved config files and processing tasks. Anytime permissions are available, a line matching the pattern user/group permissions (octal) (Change) is displayed on the screen. The 'Change' href link only appears when the current logged user is the owner of the data. For example, a shopping cart item with the line monnerville/Terapix rw-r----- (600) (Change) means I'm the owner (thus I can change the permissions), I can read/write my data, members of the Terapix group can only read (in this case use) this shopping cart item. Finally, no permission is granted for the rest of the world. The 'Preferences' top-level menu now comes with a new 'Default Permissions' entry that allows to change the default permissions applied to data. http://youpix.iap.fr/youpi/preferences/ Now that we're done with Permissions, Mathias M. and greg are working on fixing the bugs (in bugzilla). Currently Mathias M. is working on the possibility to parse a custom directory (bug #122) on the cluster in order to ingest all the configuration files automatically, to prevent Yulia (and others) to add more than 800 conf files manually... This feature should be available at the end of the week (see http://youpix.iap.fr/bugzilla/show_bug.cgi?id=122). Youpi production: The Youpi's production site has moved to youpix.iap.fr and is available at http://youpix.iap.fr/youpi and is running the latest version of the code (trunk) with permissions support. Yulia is starting to use it for T0006 ingestions. {{VI. Developpement processing: Patrick}} ------------------------------------------------------------------------- {{VII. Hardware/system/cluster Fred+ALL:}} ------------------------------------------------------------------------- - Points to discuss on hardware: .. Barney' desktop+laptop .. Catherine Grenet's desktop .. Yannick yannix2 .. fcix6 .. new mix's .. keep travel money for Japan - mix19 was rebooted once again. Concerns about the current instability of this node. - note: Patrick's scripts do not function correctly on the new nodes. Is this a problem for Fred or a problem for Patrick? {{VIII. Divers (All):}} ------------------------------------------------------------------------- Mathias S. will change the dates of his stay at IAP as Yuliana will be away in June. He will be at IAP between 13-18 July and will try to come again in August (have to look for free IAP rooms)