[Clas_offline] minutes from OLTWG meeting Nov 12, 2009
Eugene Pasyuk
pasyuk at jlab.org
Mon Nov 16 20:16:48 EST 2009
There seems to be several programs for ec calibrations.
What I see:
ec_calib
ecGammaCal
ec_photnCal
There is also ec_timing. I know for sure this one must be ported.
I suppose EC experts out there should tell us which ones are to be ported.
I think the last time pid_mon was used in g6c cooking.
We also should copy ec1_calib. This is for LAC calibration.
Perhaps IC folks have something for IC calibrations to be copied.
The same for RTPC.
I forgot to mention filter_tagr utilitie. It's part of gsim/gpp suite.
We also need to port DC alignment utilities. They are in calib/dc_cal
And DC efficiency/dead wire utilities. They are in pdu directory
-Eugene
Craig Bookwalter wrote:
> Hi Eugene,
> sc_mon was ready, I just forgot to add it to the repository. It's in
> there now.
>
> I built ecGammaCal but not ec_calib, mostly because I don't clearly
> understand ec_calib's role or source code structure--it also has its own
> versions of libsc.a and libec.a. That might be a mistake but I wasn't
> sure. I suppose ec_calib is used for electron running...?
> Should I also include pid_mon?
>
> Of the unfamiliar programs you mention, the ones in camelCase are
> related to ecGammaCal--I don't know how that program works but I guess
> it has like 5 executables. `skim' and the `view*' programs are all
> g12-specific, and can probably be relegated to on-demand builds. gif2C
> is unfamiliar to me as well. part2gamp and vice-versa can easily be
> relegated as well.
>
> Thanks for the quick feedback.
>
> --cb
>
> Eugene Pasyuk wrote:
>> Craig, Great progress!
>>
>> In addition to cc calibrations you mentioned also missing but
>> essential for calibrations and must be in default build:
>> sc_mon
>> gpid_mon
>> sync
>> ec calibrations
>>
>> I agree, some of the packages we have may be left for building on
>> demand but should be ported to svn. I'll take a look and let you know
>> what appears to be necessary to port to svn.
>>
>> Also, some of the executables in your default build do not look familiar:
>>
>> checkBetaNeutrals
>> computeCalibConsts
>> countgamp
>> filterGammas
>> fitTimeDifference
>> gamp2part
>> getCalibConsts
>> gif2C
>> part2gamp
>> partMaker
>> putConstsInMap
>> rejectGammas
>> skim
>> view-kaon
>> view-st
>> view-tag
>> view-tracks
>>
>> What are they? Are they of general interest?
>>
>>
>> One more thing just occurred to me. We have tools directory in CVS
>> with collection of scripts for various things. Particularly important
>> are scripts for caldb access. Are those ported to svn?
>> The checked out version is in /group/clas/tools There is also
>> environment CLAS_TOOLS pointing to this location.
>>
>>
>>
>> Craig Bookwalter wrote:
>>
>>> Folks,
>>> The current status of the SVN repository is as follows.
>>>
>>> A default build following the instructions here:
>>> http://clasweb.jlab.org/wiki/index.php/Building_CLAS_libraries_with_SCons_on_ifarm
>>>
>>>
>>> produces these executables:
>>> a1c
>>> bosdump
>>> ced
>>> checkBetaNeutrals
>>> computeCalibConsts
>>> countbos
>>> countgamp
>>> dc3
>>> filterGammas
>>> fitTimeDifference
>>> g12Anal
>>> g12skim
>>> gamecock
>>> gamp2part
>>> getCalibConsts
>>> gflux
>>> gif2C
>>> gmean_cooked
>>> gpp
>>> gsim_bat
>>> gsim_int
>>> make_tw_histos
>>> p2p_delay_el
>>> part2gamp
>>> partMaker
>>> putConstsInMap
>>> rejectGammas
>>> skim
>>> stn_calib
>>> tagraw
>>> tdc_calib
>>> tof_calib
>>> trk_mon
>>> trk_mon_lite
>>> user_ana
>>> veff_calib
>>> view-kaon
>>> view-st
>>> view-tag
>>> view-tracks
>>>
>>> Notable exceptions are:
>>> - the CC calibration program, which requires Qt. I'm going to defer
>>> to Maurizio for this one, because it requires an optional SCons
>>> package that I'm not familiar with.
>>> - the TPhotTiming tool for doing TOF paddle-to-paddle alignments for
>>> photon runs. I'm currently attempting to build ClasTool in order to
>>> get TPhotTiming working, but the build of ClasTool is something of a
>>> novelty under SCons at the moment. Are there any other standard CLAS
>>> packages that use ClasTool?
>>>
>>> In addition, Eugene noted a number of programs that were not included
>>> previously by default, and still are not included in the latest
>>> build. In the past many of these programs were built on demand by the
>>> user as they were needed, and this functionality still exists in
>>> SCons. For example, in analysis/bos2ntuple live some ntuple makers
>>> for a couple of very specific photoproduction channels. These should
>>> not be built by default because they are not widely useful for our
>>> collaborators. A user who wants to build such a program can just `cd'
>>> to the appropriate directory and run `scons install' from there.
>>>
>>> Of course, there are still programs that may be necessary for
>>> rungroups that do not exist yet in SVN, or exist in SVN but do not
>>> have a functional sconstruct. I'd like to know what these programs
>>> are so I can bring them into the fold and help smooth the process of
>>> experimenting with (and hopefully transitioning to) the new build
>>> system. I would especially like to incorporate missing programs in
>>> demand by the most active rungroups, like eg6, e1-dvcs and eg1-dvcs,
>>> and g9b.
>>>
>>> Comments and concerns are welcome.
>>>
>>> --cb
>>>
>>> Eugene Pasyuk wrote:
>>>
>>>> I don't have statistics for svn for svn.
>>>> I am sure it is possible, but i don't know yet how to get this
>>>> information from svn. Our svn experts can easily get this
>>>> statistics, I suppose.
>>>> My wild guess wold be that there were more changes in svn over this
>>>> period of time.
>>>>
>>>> -Eugene
>>>>
>>>>
>>>> Stepan Stepanyan wrote:
>>>>
>>>>> Eugene,
>>>>>
>>>>> Interesting statistics. Do you have any thing similar for svn
>>>>> for the past 2 months?
>>>>>
>>>>> Stepan
>>>>>
>>>>> Eugene Pasyuk wrote:
>>>>>
>>>>>> Here are some statistics of cvs activity. I looked over the same
>>>>>> period
>>>>>> of time from September 16 till today. All the information is from
>>>>>> commitlog file which keeps all commits done in repository since
>>>>>> day one,
>>>>>> May 12, 1997. It has 42186 records total.
>>>>>>
>>>>>> So, since September 16 2009:
>>>>>>
>>>>>> Total number of commits 66.
>>>>>> The majority of the changes are for Current Run web page.
>>>>>> There are 18 commits related to software.
>>>>>> Out of these 12 are in packages ported to svn.
>>>>>> Details for these 12:
>>>>>>
>>>>>> 1 commit in ClasTool. (Sep 28)
>>>>>>
>>>>>> 5 commits in packages/utilities/sync. (Oct 9) They were done in
>>>>>> one day,
>>>>>> almost simultaneously. Can count as 1 commit.
>>>>>>
>>>>>> 5 commits in packages/eloss Sep30 Oct2 Oct22 Oct 23 Oct 29
>>>>>> There is ongoing development of this package for FROST target.
>>>>>>
>>>>>> 1 commit in packages/seb done yesterday.
>>>>>>
>>>>>> These are all changes over the last two months.
>>>>>>
>>>>>> -Eugene
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, 2009-11-13 at 09:03 +0100, Marco Ripani wrote:
>>>>>>
>>>>>>
>>>>>>> Hi Stepan,
>>>>>>> yes, it is necessary to put more efforts to fix everything and
>>>>>>> then take an action. More help is needed to port the missing
>>>>>>> packages and test that they build and work fine (there are still
>>>>>>> many, as we see from Eugene's new test).
>>>>>>> But currently there is also the problem of keeping CVS and SVN in
>>>>>>> sync for packages that have been ported (e.g. user_ana). Johann
>>>>>>> is doing that using scripts but he also has to do some manual
>>>>>>> adjustments. To avoid this, the sooner we start using packages in
>>>>>>> the new system, the better.
>>>>>>>
>>>>>>> Marco
>>>>>>>
>>>>>>> Stepan Stepanyan wrote:
>>>>>>>
>>>>>>>> Hi Marco,
>>>>>>>>
>>>>>>>> I am not sure how to read your email. What do you mean "freeze
>>>>>>>> CVS asap, not today".
>>>>>>>> At the software meeting couple of month ago I though we came to
>>>>>>>> agreement that we
>>>>>>>> will discuss cvs vs svn after some experience will be gained.
>>>>>>>> Several groups expressed
>>>>>>>> interest to test svn/scon (e1-dvcs, eg1-dvcs, frost ...). So far
>>>>>>>> only Eugene made an attempt
>>>>>>>> to use svn and scon and as I can see from his email he succeeded
>>>>>>>> using the staff that
>>>>>>>> was polished by g12 for about a year, but there are still things
>>>>>>>> that either missing or not
>>>>>>>> ready yet. Why not put more efforts to fix every thing and then
>>>>>>>> take an action?
>>>>>>>>
>>>>>>>> Stepan
>>>>>>>>
>>>>>>>> Marco Ripani wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> need to freeze CVS but asap, not today. Need to first figure
>>>>>>>>> out what the best time is, with minimum impact on the other run
>>>>>>>>> groups, in particular due to the missing packages.
>>>>>>>>>
>>>>>>>>> Marco
>>>>>>>>>
>>>>>>>>> Johann Goetz wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thursday, Nov 12, 2009: SCons Build System and CLAS 6 Software
>>>>>>>>>> 12:00 pm Eastern
>>>>>>>>>> Cebaf Center A-110
>>>>>>>>>> Call in information: Dial 1-877-643-6951
>>>>>>>>>> International access: 1-302-607-2017
>>>>>>>>>> Passcode: 24630562#
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> RPMs, yum, and all that (Maurizio)
>>>>>>>>>> * creating RPM (installable pre-compiled packages with
>>>>>>>>>> dependency
>>>>>>>>>> tree) of the CLAS software is a great idea.
>>>>>>>>>> * these RPMs can served with a JLab yum server so that
>>>>>>>>>> installation on a Fedora or Red Hat (or CentOS) system
>>>>>>>>>> could be
>>>>>>>>>> done with a line like:
>>>>>>>>>> yum install clas6-reconstruction clas6-generators etc...
>>>>>>>>>> * CLAS12 has already implemented this on several flavors
>>>>>>>>>> of Linux
>>>>>>>>>> *
>>>>>>>>>> Eugene's Review of SCons Build System
>>>>>>>>>> * Generally favourable review. Eugene points out that
>>>>>>>>>> there are
>>>>>>>>>> many packages missing that are needed.
>>>>>>>>>> * Many packages will have to copied over directories
>>>>>>>>>> from CVS. * sconstruct files will have to be made.
>>>>>>>>>> * gamecock (TOF calib program) has some problems
>>>>>>>>>> (ROOT/GUI
>>>>>>>>>> related)
>>>>>>>>>> * CLASTool should be fairly easy to get to work
>>>>>>>>>> now that
>>>>>>>>>> ROOT dictionaries can be built properly.
>>>>>>>>>> From: Eugene Pasyuk <pasyuk at jlab.org>
>>>>>>>>>> To: CLAS Offline <clas_offline at jlab.org>
>>>>>>>>>> Subject: [Clas_offline] testing svn/scons
>>>>>>>>>> Date: Wed, 16 Sep 2009 10:29:31 -0400 (07:29 PDT)
>>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I tried new software tools today on ifarm. Here are some first
>>>>>>>>>> impressions about this experience.
>>>>>>>>>>
>>>>>>>>>> I followed prescription given on the Wiki:
>>>>>>>>>> http://clasweb.jlab.org/wiki/index.php/Building_CLAS_libraries_with_SCons_on_ifarm
>>>>>>>>>>
>>>>>>>>>> It took 20 minutes get the entire source tree source from svn.
>>>>>>>>>> Then I tried to build. The very first attempt failed. I
>>>>>>>>>> figured out that
>>>>>>>>>> the problem was in suggested environment setup file. I
>>>>>>>>>> modified a couple
>>>>>>>>>> of lines in it (2 minutes to figure out and fix). After that I
>>>>>>>>>> made a
>>>>>>>>>> second attempt to run scons. This time it worked. It took about 3
>>>>>>>>>> minutes to build all clas libraries and a just few executables by
>>>>>>>>>> default:
>>>>>>>>>> a1c countbos g12Anal gsim_bat tagraw view-kaon
>>>>>>>>>> view-tag
>>>>>>>>>> bosdump g12skim skim user_ana view-st view-tracks
>>>>>>>>>>
>>>>>>>>>> Out of these only few (a1c, gsim_bat, countbos, bosdump,
>>>>>>>>>> user_ana and
>>>>>>>>>> tagraw) are of interest to general CLAS user. The rest seem to
>>>>>>>>>> be g12
>>>>>>>>>> specific.
>>>>>>>>>> After doing some archeology in sconstruct files for about a
>>>>>>>>>> half an hour
>>>>>>>>>> and making some modifications to them here and there (I have zero
>>>>>>>>>> knowledge of scons or python at the moment) I managed to build
>>>>>>>>>> a few
>>>>>>>>>> more utilities, particularly DC calibration tools trk_mon,
>>>>>>>>>> trk_mon_lite
>>>>>>>>>> and dc3. I failed to build start counter calibration code. I
>>>>>>>>>> was able to
>>>>>>>>>> build fdump. gflux would not build because of the compiler
>>>>>>>>>> errors. I
>>>>>>>>>> wonder how g12 managed to build it in new build scheme from
>>>>>>>>>> the source
>>>>>>>>>> in svn. ced would not build because of the compiler errors too.
>>>>>>>>>> TOF calibration source code is in svn, but it was not modified
>>>>>>>>>> for scons
>>>>>>>>>> at all. It was imported as is from cvs with all Makefiles
>>>>>>>>>> which will not
>>>>>>>>>> work because of the different tree structure. sc_mon is
>>>>>>>>>> missing. As was
>>>>>>>>>> mentioned at the meeting ec timing calibration utilities are not
>>>>>>>>>> included as several other general purpose utilities. For some
>>>>>>>>>> reason
>>>>>>>>>> gsim_int and gpp are not included.
>>>>>>>>>>
>>>>>>>>>> To summarize, I succeeded in building most of the libraries
>>>>>>>>>> and a few
>>>>>>>>>> executables. Quite a few tools are missing (gsim_int, gpp, some
>>>>>>>>>> calibration tools). Some programs which build fine with make
>>>>>>>>>> would not
>>>>>>>>>> build with scons (gflux, ced ...?) I don't know if those
>>>>>>>>>> executables I
>>>>>>>>>> built are working, I had not had a chance to test them yet. I
>>>>>>>>>> have not
>>>>>>>>>> tried to build any of the event generators with scons.
>>>>>>>>>> By the way, there are only three of them in svn, while we have
>>>>>>>>>> more in
>>>>>>>>>> cvs.
>>>>>>>>>>
>>>>>>>>>> All in all, svn works just fine, apart from having different
>>>>>>>>>> directory
>>>>>>>>>> structure. scons works for some pieces of the software and
>>>>>>>>>> does not for
>>>>>>>>>> some
>>>>>>>>>> others. Perhaps it is not quite ready for the prime time yet.
>>>>>>>>>> Some essential
>>>>>>>>>> utilities are not working yet in new build. -Eugene
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What We Should Do Now
>>>>>>>>>> * Hire some undergraduates to move packages from CVS to
>>>>>>>>>> SVN and
>>>>>>>>>> add sconstruct files to build these packages.
>>>>>>>>>> * Armand Fraiser was a big help initially with the move to
>>>>>>>>>> SVN/SCons and he could continue to do so.
>>>>>>>>>> * There is some detailed help information already, it
>>>>>>>>>> probably
>>>>>>>>>> needs a bit of reorganization on the wiki.
>>>>>>>>>> * Should we have a list of programs/libraries that
>>>>>>>>>> should be moved
>>>>>>>>>> from CVS to SVN (Armand already created an spreadsheet
>>>>>>>>>> that has
>>>>>>>>>> some of this information. It will be sent to Craig and
>>>>>>>>>> he will
>>>>>>>>>> look into it.
>>>>>>>>>> * We need to freeze CVS today.
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Clas_offline mailing list
>>>>>>>>>> Clas_offline at jlab.org
>>>>>>>>>> https://mailman.jlab.org/mailman/listinfo/clas_offline
>>>>>>>>>>
>>>>>>>>>
>>>>>> _______________________________________________
>>>>>> Clas_offline mailing list
>>>>>> Clas_offline at jlab.org
>>>>>> https://mailman.jlab.org/mailman/listinfo/clas_offline
>>>>>>
>>>> _______________________________________________
>>>> Clas_offline mailing list
>>>> Clas_offline at jlab.org
>>>> https://mailman.jlab.org/mailman/listinfo/clas_offline
>>>>
>>>
>> _______________________________________________
>> Clas_offline mailing list
>> Clas_offline at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/clas_offline
>>
>
>
More information about the Clas_offline
mailing list