<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Hi Rafo,</div><div id="AppleMailSignature">I suggest memory profiling to see if GC is the reason of the performance degradation, or quick check would be to set -Xms=-Xmx=someLargeValue and see if you get some speed up.</div><div id="AppleMailSignature">-vardan <br><br>Sent from my iPhone</div><div><br>On May 21, 2017, at 12:54 AM, Rafayel Paremuzyan <<a href="mailto:rafopar@jlab.org">rafopar@jlab.org</a>> wrote:<br><br></div><blockquote type="cite"><div>
<font size="+1">Hi Alessandra, Norman, all<br>
<br>
thank you for replay and your tests.<br>
<br>
I tested both 2015 and 2016 data using v4-4 detector on UNH
computers.<br>
I have use 3.8 JAR (the jar for 2015 pass6). 3.9 JAR (the jar for
2016 pass0 recon), and the NEW jar v051717 (the newest jar tag is
v051717)<br>
<br>
Ok, I also noticed that recon of 2015 data is faster that 2016
data. <br>
Also seems the new jar is 20% slower than the 3.9 jar for 2016
data, and about 60% slower for 2015 data.<br>
now recon speed is about 2.55 EV/S for 2015 data, This is too slow<br>
it cause more than 40h for a single file.<br>
<br>
Ths is summary of code speeds with different jar files</font><br>
<table cellspacing="0" border="0">
<colgroup width="214"></colgroup> <colgroup width="266"></colgroup>
<colgroup width="301"></colgroup> <colgroup width="294"></colgroup>
<tbody>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" colspan="4" valign="middle" height="25" align="center"><font size="4" face="Sylfaen">V4-4 Detector,
UNH (endeavour), 5K events are reconstructed</font></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" valign="middle" height="25" align="center"><font size="4" face="Sylfaen"><br>
</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="left"><font size="4" face="Sylfaen">Events per second</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="left"><font size="4" face="Sylfaen">Events per second</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="left"><font size="4" face="Sylfaen">Events per second</font></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" height="25" align="left"><font size="4" face="Sylfaen"><br>
</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="left"><font size="4" face="Sylfaen">3.8JAR (2015 recon jar)</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="left"><font size="4" face="Sylfaen">3.9JAR, 2016 pass0 recon jar</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="left"><font size="4" face="Sylfaen">v051717JAR, jar vor tpass1</font></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" height="25" align="left"><font size="4" face="Sylfaen">2015 Data 5772, file 20</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="right"><font size="4" face="Sylfaen">5.07</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="right"><font size="4" face="Sylfaen">5.19</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="right"><font size="4" face="Sylfaen">3.157</font></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" height="25" align="left"><font size="4" face="Sylfaen">2016 Data file 25</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="left"><font size="4" face="Sylfaen"><br>
</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="right"><font size="4" face="Sylfaen">3.11</font></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000; border-right:
1px solid #000000" align="right"><font size="4" face="Sylfaen">2.53</font></td>
</tr>
</tbody>
</table>
<font size="+1"><br>
<br>
*However* I looked into job Wall times for pass0 recon.<br>
The recon speed is more than 7.4 Events/sec, which is about x3
faster than with the new JAR.<br>
<br>
I again checked *same 3.9 jar*, which is slower again.<br>
I don't know why now the code speed is so low!<br>
<br>
<br>
Norman, I have tried the "-DdisableSvtAlignmentConstants", but it
didn't work<br>
<br>
=================The command===============<br>
java -XX:+UseSerialGC -cp hps-distribution-3.11-v051717-bin.jar
org.hps.evio.EvioToLcio -x
/org/hps/steering/recon/PhysicsRun2016FullRecon.lcsim -r -d
HPS-PhysicsRun2016-v5-3-fieldmap_globalAlign -R 7796
-DoutputFile=out_7796_0 -DdisableSvtAlignmentConstants
hps_007796.evio.25 -n 10000<br>
<br>
============The error backtrache============<br>
2017-05-21 00:45:39 [CONFIG] org.hps.evio.EvioToLcio parse ::
using steering resource
/org/hps/steering/recon/PhysicsRun2016FullRecon.lcsim<br>
2017-05-21 00:45:39 [CONFIG] org.hps.evio.EvioToLcio parse :: set
max events to 10000<br>
2017-05-21 00:45:48 [INFO] org.hps.rundb.RunManager <init>
:: ConnectionParameters { database: hps_run_db_v2, hostname:
<a href="http://hpsdb.jlab.org">hpsdb.jlab.org</a>, password: darkphoton, port: 3306, user: hpsuser }<br>
2017-05-21 00:45:48 [CONFIG] org.lcsim.job.JobControlManager
addVariableDefinition :: outputFile = out_7796_0<br>
2017-05-21 00:45:48 [CONFIG] org.hps.evio.EvioToLcio parse :: set
steering variable: outputFile=out_7796_0<br>
2017-05-21 00:45:48 [SEVERE] org.hps.evio.EvioToLcio parse :: bad
variable format: disableSvtAlignmentConstants<br>
java.lang.IllegalArgumentException: Bad variable format:
disableSvtAlignmentConstants<br>
at org.hps.evio.EvioToLcio.parse(EvioToLcio.java:393)<br>
at org.hps.evio.EvioToLcio.main(EvioToLcio.java:97)<br>
<br>
Exception in thread "main" java.lang.IllegalArgumentException: Bad
variable format: disableSvtAlignmentConstants<br>
at org.hps.evio.EvioToLcio.parse(EvioToLcio.java:393)<br>
at org.hps.evio.EvioToLcio.main(EvioToLcio.java:97)<br>
<br>
Rafo<br>
</font><br>
<br>
<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/20/2017 06:17 AM, Alessandra
Filippi wrote:<br>
</div>
<blockquote type="cite" cite="mid:alpine.LRH.2.00.1705201201150.23923@to05xl.to.infn.it">Hi
Rafo, all,
<br>
I also noticed that the reconstruction for 2016 data is about
twice as slow as compared to 2015 (whichever geometry and
reconstruction version).
<br>
This happens when I run the aligned geometry as well as the
"current" one (v5.0), and the geometry taken from the db as well
(the result is the same as v5.0). I did not make any test with
v4.4, though - actually as far as svt alignment is concerned it
should be the same as v5.0. Can you please try and make the same
short test with the newest jar with v4.4?
<br>
This happens to me both with hps-java 5.10 and 5.11 (not the most
updated one).
<br>
<br>
I would be surprised if it could be something connected to the
alignment, unless for some reason new positions and harder tracks
trigger some long loops in the reconstruction. But this happens
(to me) also with the
<br>
standard geometry, so a check with the one used with pass0 (that
should however be equivalent to v5.0) could at least help to rule
out, or blame on, the alignment step.
<br>
Thanks, cheers
<br>
Alessandra
<br>
<br>
<br>
ps. make also sure that the correct fieldmap is called in all the
compact files - you never know!
<br>
<br>
<br>
<br>
On Fri, 19 May 2017, Rafayel Paremuzyan wrote:
<br>
<br>
<blockquote type="cite">Hi All,
<br>
<br>
During the testing the recon for test pass1,
<br>
I noticed the recon time is more than x2 longer wrt pass0 recon
time.
<br>
<br>
To demonstrate it
<br>
I submit 3 simple jobs with 10K events to reconstruct, with new
pass1 xml
<br>
file (this has the new jar v051717, and the new detector
<br>
HPS-PhysicsRun2016-v5-3-fieldmap_globalAlign),
<br>
and the old pass0 xml file (pass0 jar release 3.9, and the
detector
<br>
HPS-PhysicsRun2016-Nominal-v4-4-fieldmap)
<br>
<br>
Below is a printout from the jobs with a new JAR, v051717, the
average time
<br>
for 1000 events is more than 7 minutes
<br>
===================== LOG from the v051717 JAR
<br>
==============================
<br>
2017-05-19 09:36:51 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10614074 with sequence 0
<br>
2017-05-19 09:43:13 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10615074 with sequence 1000
<br>
2017-05-19 09:49:18 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10616074 with sequence 2000
<br>
2017-05-19 09:55:54 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10617074 with sequence 3000
<br>
2017-05-19 10:02:55 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10618074 with sequence 4000
<br>
2017-05-19 10:09:57 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10619074 with sequence 5000
<br>
2017-05-19 10:16:13 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10620074 with sequence 6000
<br>
2017-05-19 10:25:20 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10621074 with sequence 7000
<br>
2017-05-19 10:32:56 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10622074 with sequence 8000
<br>
2017-05-19 10:36:19 [WARNING]
org.hps.recon.tracking.TrackerReconDriver
<br>
process :: Discarding track with bad HelicalTrackHit (correction
distance
<br>
0.000000, chisq penalty 0.000000)
<br>
2017-05-19 10:42:03 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10623074 with sequence 9000
<br>
2017-05-19 10:47:44 [INFO] org.hps.evio.EvioToLcio run ::
maxEvents 10000
<br>
was reached
<br>
2017-05-19 10:47:44 [INFO] org.lcsim.job.EventMarkerDriver
endOfData ::
<br>
10000 events processed in job.
<br>
2017-05-19 10:47:44 [INFO] org.hps.evio.EvioToLcio run :: Job
finished
<br>
successfully!
<br>
<br>
<br>
And below is the Job log info from the pass0 jar. The average
time for 1000
<br>
events is less than 3 minutes
<br>
===================== LOG from the 3.9 release JAR
<br>
==============================
<br>
2017-05-19 13:19:46 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10614074 with sequence 0
<br>
2017-05-19 13:23:36 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10615074 with sequence 1000
<br>
2017-05-19 13:27:03 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10616074 with sequence 2000
<br>
2017-05-19 13:30:40 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10617074 with sequence 3000
<br>
2017-05-19 13:34:20 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10618074 with sequence 4000
<br>
2017-05-19 13:38:11 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10619074 with sequence 5000
<br>
2017-05-19 13:41:43 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10620074 with sequence 6000
<br>
2017-05-19 13:45:54 [WARNING]
org.hps.recon.tracking.TrackerReconDriver
<br>
process :: Discarding track with bad HelicalTrackHit (correction
distance
<br>
0.000000, chisq penalty 0.000000)
<br>
2017-05-19 13:46:05 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10621074 with sequence 7000
<br>
2017-05-19 13:50:08 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10622074 with sequence 8000
<br>
2017-05-19 13:55:03 [INFO] org.lcsim.job.EventMarkerDriver
process :: Event
<br>
10623074 with sequence 9000
<br>
2017-05-19 13:58:27 [INFO] org.hps.evio.EvioToLcio run ::
maxEvents 10000
<br>
was reached
<br>
2017-05-19 13:58:27 [INFO] org.lcsim.job.EventMarkerDriver
endOfData ::
<br>
10000 events processed in job.
<br>
2017-05-19 13:58:27 [INFO] org.hps.evio.EvioToLcio run :: Job
finished
<br>
successfully!
<br>
<br>
I also tried to do reconstruction by myself interactively, but I
am getting
<br>
error below,
<br>
<br>
The command
<br>
/apps/scicomp/java/jdk1.7/bin/java -XX:+UseSerialGC -cp
<br>
hps-distribution-3.9-bin.jar org.hps.evio.EvioToLcio -x
<br>
/org/hps/steering/recon/PhysicsRun2016FullRecon.lcsim -r -d
<br>
HPS-PhysicsRun2016-v5-3-fieldmap_globalAlign -R 7796
-DoutputFile=out_7796_0
<br>
hps_007796.evio.0 -n 10000
<br>
<br>
The Error traceback
<br>
017-05-19 14:58:44 [CONFIG] org.hps.evio.EvioToLcio parse ::
using steering
<br>
resource /org/hps/steering/recon/PhysicsRun2016FullRecon.lcsim
<br>
2017-05-19 14:58:44 [CONFIG] org.hps.evio.EvioToLcio parse ::
set max events
<br>
to 10000
<br>
2017-05-19 14:58:45 [CONFIG] org.lcsim.job.JobControlManager
<br>
addVariableDefinition :: outputFile = out_7796_0
<br>
2017-05-19 14:58:45 [CONFIG] org.hps.evio.EvioToLcio parse ::
set steering
<br>
variable: outputFile=out_7796_0
<br>
2017-05-19 14:58:45 [CONFIG] org.lcsim.job.JobControlManager
initializeLoop
<br>
:: initializing LCSim loop
<br>
2017-05-19 14:58:45 [CONFIG] org.lcsim.job.JobControlManager
initializeLoop
<br>
:: Event marker printing disabled.
<br>
2017-05-19 14:58:45 [INFO]
<br>
org.hps.conditions.database.DatabaseConditionsManager
resetInstance ::
<br>
DatabaseConditionsManager instance is reset
<br>
Exception in thread "main" java.lang.UnsatisfiedLinkError:
<br>
/u/apps/scicomp/java/jdk1.7.0_75/jre/lib/i386/xawt/libmawt.so:
libXext.so.6:
<br>
cannot open shared object file: No such file or directory
<br>
at java.lang.ClassLoader$NativeLibrary.load(Native
Method)
<br>
at
java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1965)
<br>
at
java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1890)
<br>
at
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1851)
<br>
at java.lang.Runtime.load0(Runtime.java:795)
<br>
at java.lang.System.load(System.java:1062)
<br>
at java.lang.ClassLoader$NativeLibrary.load(Native
Method)
<br>
at
java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1965)
<br>
at
java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1890)
<br>
at
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1872)
<br>
at java.lang.Runtime.loadLibrary0(Runtime.java:849)
<br>
at java.lang.System.loadLibrary(System.java:1088)
<br>
at
<br>
sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:67)
<br>
at
<br>
sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:47)
<br>
at java.security.AccessController.doPrivileged(Native
Method)
<br>
at java.awt.Toolkit.loadLibraries(Toolkit.java:1653)
<br>
at java.awt.Toolkit.<clinit>(Toolkit.java:1682)
<br>
at java.awt.Component.<clinit>(Component.java:595)
<br>
at org.lcsim.util.aida.AIDA.<init>(AIDA.java:68)
<br>
at
org.lcsim.util.aida.AIDA.defaultInstance(AIDA.java:53)
<br>
at
org.hps.evio.RfFitterDriver.<init>(RfFitterDriver.java:31)
<br>
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
<br>
Method)
<br>
atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce<br>
ssorImpl.java:57)
<br>
atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru<br>
ctorAccessorImpl.java:45)
<br>
at
java.lang.reflect.Constructor.newInstance(Constructor.java:526)
<br>
at java.lang.Class.newInstance(Class.java:379)
<br>
at
<br>
org.lcsim.job.JobControlManager.setupDrivers(JobControlManager.java:1199)
<br>
at
org.hps.job.JobManager.setupDrivers(JobManager.java:82)
<br>
at
<br>
org.lcsim.job.JobControlManager.setup(JobControlManager.java:1052)
<br>
at
<br>
org.lcsim.job.JobControlManager.setup(JobControlManager.java:1110)
<br>
at org.hps.evio.EvioToLcio.parse(EvioToLcio.java:407)
<br>
at org.hps.evio.EvioToLcio.main(EvioToLcio.java:97)
<br>
<br>
<br>
<br>
I see this library libXext.so.6: in /usr/lib64, but not in
/usr/lib,
<br>
when I put /usr/lib64 in my LD_LIBRARY_PATH, then it complaines
again (see
<br>
below)
<br>
<br>
Exception in thread "main" java.lang.UnsatisfiedLinkError:
<br>
/u/apps/scicomp/java/jdk1.7.0_75/jre/lib/i386/xawt/libmawt.so:
libXext.so.6:
<br>
wrong ELF class: ELFCLASS64
<br>
<br>
I would appreciate, if I get some help on running the
reconstruction
<br>
interactively, then I could look more closely into logs
<br>
of the old, and new JAR files.
<br>
<br>
Rafo
<br>
<br>
<br>
____________________________________________________________________________
<br>
<br>
Use REPLY-ALL to reply to list
<br>
<br>
To unsubscribe from the HPS-SOFTWARE list, click the following
link:
<br>
<a class="moz-txt-link-freetext" href="https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DHPS-2DSOFTWARE-26A-3D1&d=DwIDaQ&c=lz9TcOasaINaaC3U7FbMev2lsutwpI4--09aP8Lu18s&r=0HDJrGO9TZQTE97J9Abt2A&m=xnbGP5VHYWRAQRWRksVgMnYvBkXWI4roLxztdJ0Tp9I&s=ppNYedSrn5DPaIZZJgRZu8tBDeSjroqbj_PoevFoFpI&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DHPS-2DSOFTWARE-26A-3D1&d=DwIDaQ&c=lz9TcOasaINaaC3U7FbMev2lsutwpI4--09aP8Lu18s&r=0HDJrGO9TZQTE97J9Abt2A&m=xnbGP5VHYWRAQRWRksVgMnYvBkXWI4roLxztdJ0Tp9I&s=ppNYedSrn5DPaIZZJgRZu8tBDeSjroqbj_PoevFoFpI&e=</a>
<br>
<br>
<br>
</blockquote>
<br>
########################################################################
<br>
Use REPLY-ALL to reply to list
<br>
<br>
To unsubscribe from the HPS-SOFTWARE list, click the following
link:
<br>
<a class="moz-txt-link-freetext" href="https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DHPS-2DSOFTWARE-26A-3D1&d=DwIDaQ&c=lz9TcOasaINaaC3U7FbMev2lsutwpI4--09aP8Lu18s&r=0HDJrGO9TZQTE97J9Abt2A&m=xnbGP5VHYWRAQRWRksVgMnYvBkXWI4roLxztdJ0Tp9I&s=ppNYedSrn5DPaIZZJgRZu8tBDeSjroqbj_PoevFoFpI&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DHPS-2DSOFTWARE-26A-3D1&d=DwIDaQ&c=lz9TcOasaINaaC3U7FbMev2lsutwpI4--09aP8Lu18s&r=0HDJrGO9TZQTE97J9Abt2A&m=xnbGP5VHYWRAQRWRksVgMnYvBkXWI4roLxztdJ0Tp9I&s=ppNYedSrn5DPaIZZJgRZu8tBDeSjroqbj_PoevFoFpI&e=</a>
</blockquote>
<br>
<br>
<hr>
<p align="left">
Use REPLY-ALL to reply to list</p>
<p align="center">To unsubscribe from the HPS-SOFTWARE list, click the following link:<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DHPS-2DSOFTWARE-26A-3D1&d=DwMDaQ&c=lz9TcOasaINaaC3U7FbMev2lsutwpI4--09aP8Lu18s&r=vQNuYt02mjBcQbsZ9w97_A&m=hS-BcSam_LWYSZhycQKKqvhFRvTf1yxEFVM3DSjfYds&s=eDfCmHt-dwCwN3vHc_eLm8BnqAsfIU2OzSXhexypRtc&e=" target="_blank">https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=HPS-SOFTWARE&A=1</a>
</p>
</div></blockquote></body></html>