Minutes for the terameeting of 21st October 2005 Present: Henry, Emmanuel, Yannick, Laurent, Chiara, Jean-Christophe, Mireille

Status T0003 (Laurent)
 Laurent distributed before the meeting a summary describing the current status of the release (included below). In principle, Mireille, Yannick, Henry can start grading the QFITs-in data.

 Unless major problems occur, STEP1 should be finished by the beginning of November. However, the new SExtractor bug prevents sending catalogues to CADC now.

 Jean-Christophe should not copy any QFITS-in data to the output bays.

SExtractor bug:

 8500 ldac catalogues were created which have problems in certain error measurements ERRA/B/THETAWIN_IMAGE are zero

 SExtractor has been patched to fix this bug, but these ldacs will need to be regenerated with the latest version of sextractor.

 We will start this process at the end of the processing.

 Emmanuel will to patch SCAMP so that it can correctly handle this kind of input

 Laurent will start making his first tests of STEP2 with SCAMP and SWarp next week

Project management and testing (all)

 Emmanuel's bug was not found until several weeks after the release was started
 How can we better monitor the output of the pipeline in order to correct problems in a shorter timescale?
 It is suggested that we have test dataset (e.g, 10 images in i* , 10 in u*) in the w1 field can be used as test data for the entire processing chain. The RunID will be changed as TERATEST.
 The results of processing these images will be compared to the previous runs to locate any potential problems
 A version of SPICA with reduced functionality will be supplied to Emmanuel for testing.

SCAMP (Emmanuel): Outstanding problems

 SCAMP doesn't handle correctly objects with zero positional errors (like those produced by the buggy version of SExtractor).

 In order to have clearer plots, Emmanuel will label the CCDs on the distortion grid plot generated by SCAMP.

 In the Tully PI data there are many, many faint sources which have small errors, and SCAMP cannot find a good astrometric solution; one possible solution is simply to raise the S/N threshold. Mireille will try this next week on Tully data.

 Management of photometric errors between images with different depths also needs to be investigated

 Managing unstable focal planes (generation of .ahead); one possible solution could be also to use several reference catalogues

 Generation of .ahead files: there are some images with wrong astrometric; perhaps we can use data from multiple filters to better constrain the astrometric solutions

WIRCAM (Chiara)

 There was a bug in SWarp which mean that the test images which were produced by Chiara were in fact not weighted. The re-processing using the right weighting scheme is under way by Chiara.

 Star-galaxy separation using selection in (J-K) colour seems to work pretty well.

 The images are very shallow, hopefully deeper data will be soon available.

Web-services (Jean-Christophe)
 A lot of work still required to make the QF web-services AVO compliant

 Yannick suggests that JC makes a trip to Strasbourg to try to resolve this problem as quickly as possible. He reminds Jean-Christophe the dead line for delivery is Nov. 15.

Mathias Schultheis (Yannick)

 Matthias will be responsible for the inter-calibration of the wide survey fields under the direction of Henry. He will start working on T0003 wide release immediately.

 Matthias will only be able to make three trips to Paris before the end of this year; these dates will be defined with Henry.

 Matthias will be added to the Teralist mailing list and will provided with an account on Terapix clister once the final agreement between IAP/Terapix and the Observatoire de Besancon is signed.

Hardware (Emmanuel, Yannick)

 The desktop machines from Agorus have still not been delivered, they should be delivered the start of next week. They are very slow!

 The motherboard of mix7 is still not ready, and we have no definitive delivery date for this machine;

 Several disk failures: 250-GB Serial-ATA (mix1); brownie (fc3/fc4) 250 GB maxtor also failed

 Not sure if we have the delivery records for these disks

 mix2 had to be rebooted yesterday probably due to xfs error (according to the dmesg)

 mix5 has crashed once during a SWarp

 mix4 has has crashed twice in the last month

Astro-wise (Emmanuel, Yannick)

 Anis should start the 4th November; try to arrange a meeting on Thursday with Emmanuel and Yannick

 15, 16, 17 of November there is an Astro-Wise workshop at Leiden

 Emmanuel will be there on 15th, 16th (astrometric solutions); Yannick will be present 17th

 Valentijn has proposed to give an Oracle licence to TERAPIX in return for processing data; we would need to dedicate a TERAPIX machine to this processing. This would also need a standard operating system like Redhat. This could be advantageous for TERAPIX (from a funding point of view) but we need to make sure we know how much work it will be...

 Next tera-meeting: Oct. 28, room 281.

 8700 images sont passees dans QFITS, soit 2000 de plus que la semaine derniere.

 sur ces 8700, presque 1000 sont des donnees PI (soit 10%).

 Le taux de QFITS est d'environ 400/jour

 Tous les pix ont quasiment termine leur QF PI (ce qui explique le taux de QF plus bas).

 Sanders est en attente de reprocessing au CFH.

 1470 images sont en attente de validation (soit 15%).

 Il reste encore 700 images dans datatransfer pour le run 04AL

 A ceci s'ajoute 750 images nouvelles du run 05BL

 En cours : mix3 (1000 images) + mix4 (650 images) + mix6 (620 images) : 2300 images

 Duree estimee de la fin des QF : 15-20 jours.

 Le temps moyen d'un QFITS sur les pix est de 50 minutes, de 20 minutes sur les mix

Anomalies detectees :

 SPICA est reste bloque sur mix3 (merci Manu) : cause inconnue.

 Arret de SPICA du au CDS non accessible => QFITS patche.

 des images ont ete transferees plusieurs fois sur plusieurs nodes differents ou sur un meme node. (ceci signifie que certains RunID sont partages sur plusieurs machines).

 en consequence, la disposition des donnees sur le cluster est sub-optimale.

 SExtractor 2.4.4 mis a jour sur le cluster par Emmanuel . Besoin d'une rustine pour recalculer tous les ldac precedemment generes.

 scp defectueux sur certains QFITS (70 sur 8000 soit 1%) : nettoyage effectue + SPICA corrige.

Previsions pour la semaine prochaine :

 Re-activer le mode reprocessing des QFITS dans SPICA

 Premier stack de test sur des donnees d'un meme RunID splittees sur plusieurs nodes (jamais teste auparavant).