Terapix summary of activities 07/04/09 - 27/04/09 - Missing summaries from Greg and Fred. Youpi testing (Mathias M. and Yuliana) Youpi tests (mainly) for SCAMP and SExtractor plugins. Some problems found with the image selector and with the multiple image ingestion. Define default and T0006 SCAMP and SExtractor config/param files. TBD: - send Bugzilla report for the image selector bug - send Bugzilla report for the multiple ingestions bug (?) [ henry: this is just a question of learning how to use BugZilla]? (From mathias -s ) - YOUPI training and we discussed several issues and found several bugs which were already fixed by Mathias M and Greg - As already pointed out by Yuliana we had several discussions about user permissions in Youpi (see her e-mail and Matthias M) -we have to decide these important issues for T0006 as soon as possible. - Stil to do: Comparison of scamp output in Youpi and command line -hope to do this tomorrow.. greg summary: YOUPI DATABASE AND MODEL table ConfigFile which handling the support of configuration file has changed A new field was added : "type" to generalise the concept of file handling through youpi Then, modifying the configfilewidget.js class with a new argument in the prototype function ConfigFileWidget(container, pluginId) MOVE TO function ConfigFileWidget(container, pluginId, filetype) A new instance of that class should be: var cf1 = new ConfigFileWidget('config_file_div', '{{ plugin.id }}', 'config') -> handling a plugin related configuration file var cf2 = new ConfigFileWidget('parameter_file_div', '{{ plugin.id }}', 'param') -> handling a plugin related parameter file Modifying all the related code to make it works . Those modifications were added to prevent using more external files in the future, for processing using, through the pipeline. Adding to the model of youpi the sex.default.param which contains a list of words(which needs to be defined for a general use of the pipeline as the default one). Adding default.nnw, sex.default.conv too. BUGS FIXING - avoid overwriting the default configurations files for all plugin - fixing two bugs in the sextractor dual image mode (loss of the paths in dual mode) WIRCAM PI processing (Patrick, Henry) : Bouvier 07BF99 : - processing finished - report written - release sent (Jerome has downloaded it). Bouvier 08AF98 (second part): - data downloaded - qfits + scamp done - first stack done - The image checking is ongoing. [ henry : we had decided that all processing other than wuds should be put on hold, and all efforts should be concentrated on WUDS] Dubus: - skysubstraction done - Check of the skysubstracted images done WUDS : - All data are on mix19 - All cubes have been sliced (around 4000 images) - The qfits are finished - Next step is the grading : to be done here or remote ?? [I thought we had agreed to do the grading after the sky-subtraction step] DEV : - documentation reading on Flex and Bison (grammar interpretors) - source code reading (E.Bertin fits libraries & missfits) WIRCAM processing (WIRDS, 07BC23) (henry) - 07BC23 data has been completely processed to final stacks. However, there are photometric offsets of around 0.2 - 0.3 mags in the final stacks with respect to 2MASS - It turns out that the reason part of the processing was not carried out correctly at CFHT is because sky-subtraction failed in loic's pipeline because the PI chose the wrong dither pattern. Loic has offered to try to reprocess the data but the for the moment henry will try to use 2MASS to carry out the calibration. Unfortunately the initial calibration with 2MASS didn't work too well; it seems to be sensitive to the cut in errors assumed. - WIRDS data release has still not happened because of problems with file-naming at CADC. Rich Bielby is sorting this out with Chris. - Rich has started WIRDS-D3 processing. Youpi software developement (from Mathias) Support for .ahead files has been implemented into the Scamp plugin. For now, Youpi's current behaviour is to halt (not allowing to add the selection to the shopping cart) on warnings when either .ldac or .ahead files are missing. I am not sure this is the best behaviour for Youpi users. I think we should discuss that point: maybe it's better to only display warnings, without blocking the process of adding the selection to the shopping cart... [note added by henry: does this mean that there has to be a .ahead file for each input file?] Concerning Youpi testing by Yulia and Mathias S., I think that we should all take some time to discuss and define Youpi user policies (or permissions). For now, Youpi does not handle user policies at all (because they are not yet defined!), preventing Yulia and Mathias S. to work together on T0006 inside Youpi. For example, when doing a Scamp processing, Youpi will automatically look for .ldac files from previously successful QualityFITS but on a per-user basis. This kind of rule does not allow to share the work between people in a team (i.e same group). Yulia will not be able to use Mathias's QualityFits results as input for Youpi's search of .ldac files. This should be possible for people belonging to a common group to share data. [this seems the right way to go: is there any reason not to do this?] There are tons of examples right now that highlights this user/group policies problems. It could take some time to implement it (because it's related to virtually every part of the application). Greg and I need clear definitions of policies for Youpi. System administration: Youpix (the dedicated server for Youpi) is online! [this machine is part of the terapix network?] The web server is installed and opened to the world (for now access is password restricted). Fred helped me to solve a network issue that prevented youpix to be accessible from the Internet. We'll have to install Django and all its dependancies (but the handbook needs improvements first). Finally, the big news is that Youpi is no more using Subversion for managing its source code. We are using Mercurial instead. The main benefits are: - distributed revision control system (Subversion is a centralized RCS). which means that users can work even when not connected to a network (I can now commit in the train!) - Every developer has a full project history locally - Very fast since no network is involved during most operations - Small learning curve for Subversion users - Written in Python (highly hackable) The current Youpi repository is available at http://youpix.iap.fr/mercurial/ (login:preview, passwd: youpiPreview) and is now the preferred way to follow Youpi development. We plan to set up an https (SSL) repository for the source code when the first release of Youpi is available (and the website is online). Some pointers: http://en.wikipedia.org/wiki/Mercurial_(software) http://hgbook.red-bean.com/read/preface.html http://en.wikipedia.org/wiki/Distributed_revision_control ****** *** TB Discussed with all during the tera-meeting *** Main issue for the moment: Youpi policies and user permissions. - Key points: (1) prevent users from overwriting work of other users (2) policies for viewing and using data of other users - define owners / groups / other users and the corresponding permissions - configuration and parameter files: (1) default ones should be write-protected (2) user-specified ones: allow only the owner to edit them, but define permissions for group and/or other to use them - ingested *p.fits images (?) (1) who can ingest images (2) who can delete images (3) disk-space quotas - TAGs and saved selections: (1) for the saved selections: ownership and user permissions (2) for the TAGs: more complicates as some of them might be common for all users (instrument, filters) or users within a group (obs. programme, obs.run, pointings), while other could be user-specific. How to deal with a combination of both types ? - image grading: (1) priorities (2) history - processing results: (1) warn group / other when already exist (2) allow group / other to use them (3) processing priorities ? - shopping card: (1) allow other users to see all current processes (?) - deletion (?): (1) Youpi admins (2) owner ---------------- From Mathias M.: ---------------- Dear all, I would like to discuss about Youpi user permissions. I think we'll need to add support for them in the next few weeks (the sooner, the better). The Django web framework we are using for Youpi already defines a set of policies for every model (database table) in use. For example, the 'tag' model we are using allows to use tags within Youpi. Each tag has a name, owner, and optional comment. Along with this 'tag' model, Django automatically creates this set of policies: youpi | tag | Can add tag youpi | tag | Can change tag youpi | tag | Can delete tag Every admin user (logged into the admin interface) can then apply any set of policies to any user. For a given user, if the line 'Can add tag' is unchecked, the user won't be able to add a new tag in Youpi. These are basic tags defined by the framework. Now, I think that Youpi users might want other kind of permissions/granularity. As I discussed briefly with Yulia, it might be a good idea to have such a custom permission: youpi | tag | Can edit others tags to specify if someone can edit (change/remove) other's tags. But what about other custom permissions (there could be a lot more)? I think it could be great if a list of custom user permissions could be defined so that we can add support for them as soon as possible. I think that the following permissions may be interesting: - Can use the QualityFits-in plugin - Can use the Sextractor plugin - Can use the Scamp plugin - Can use the Swarp plugin - Can add items to the shopping cart - Can execute jobs on the cluster ... Cheers, Mathias