[Halld-offline] scons install (in sim-recon) takes forever

[Nathan Sparks]

Richard,

By default, Scons calculates a md5 checksum of the target file's contents to determine if it has changed.
This takes longer for larger target files, such is the case when debug symbols are turned on.
Also, it will clearly take longer the more plugins you have; you probably noticed it after more plugins were added to sim-recon.

David’s pull request which was merged today reduces the average size of target files substantially, which should speed up the md5 check.

The behavior of how scons decides to rebuild can be customized but it is not clear to me if we should modify it after David’s PR.
See the documentation on the Decider at
http://www.scons.org/doc/HTML/scons-user/ch06.html#idm139837621900832 <http://www.scons.org/doc/HTML/scons-user/ch06.html#idm139837621900832>

-Nathan

> On Mar 23, 2017, at 2:05 PM, Richard Jones <richard.t.jones at uconn.edu> wrote:
> 
> Natthan,
> 
> How can it matter if I have debug on or off, if there is nothing to rebuild and nothing to do, just check that everything is up to date and return a "nothing to do" message? Up until a few weeks ago, this was very fast on my Centos 6 machine. Now it is about 20 minutes and pretty much saturates my io to the nfs server over that period. The process is "scons" (no compilers or ld or installers are ever invoked) so I cannot see what it is doing, except that it is deep in some python loop and reading very slowly through every .so and .o file in sim-recon.
> 
> -Richard J.
> 
> On Thu, Mar 23, 2017 at 10:13 AM, Nathan Sparks <nsparks at jlab.org <mailto:nsparks at jlab.org>> wrote:
> Hi Richard,
> 
> Here are some quick tests using sim-recon commit a5474aa on my (non-Farm) RHEL 7 machine, with a very light load:
> 1. If I build with debug symbols off (8.5 min. w/ 8 threads) and then run “scons -j8 install DEBUG=0” in the src directory,
> it takes ~ 15 s for scons to report back that the “install” is up to date.
> 
> 2. If I rebuild with debug symbols on (19 min. w/ 8 threads) and then run “scons -j8 install” in the src directory,
> it takes ~ 1.5 min. to output that the installation is up to date.
> 
> I having been doing my builds with DEBUG=0 for a long time now, so I am used to the quicker builds/checks.
> 
> Maybe the longer time it takes to check a debug build in combination with a heavy system load can explain your observation.
> 
> -Nathan
> 
> > On Mar 23, 2017, at 7:23 AM, Richard Jones <richard.t.jones at uconn.edu <mailto:richard.t.jones at uconn.edu>> wrote:
> >
> > Hello all,
> >
> > Did someone check in a change that might have caused this? A few weeks back, if I did "scons install" in sim-recon and there was nothing to do (no updates since last build) it would return with success and report up-to-date within less than 1 minute. Now it takes forever (I hit ctrl-C after 20 minutes) to discover that it has nothing to do.
> >
> > -Richard J.
> > _______________________________________________
> > Halld-offline mailing list
> > Halld-offline at jlab.org <mailto:Halld-offline at jlab.org>
> > https://mailman.jlab.org/mailman/listinfo/halld-offline <https://mailman.jlab.org/mailman/listinfo/halld-offline>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20170323/5677bac6/attachment-0002.html>