[Halld-offline] scons install (in sim-recon) takes forever

[Richard Jones]

Hello Nathan and all,

That made a big difference! Now I am back to completing a null build in
under 20 seconds (from 20 minutes). Thanks for that.

-Richard J.

On Thu, Mar 23, 2017 at 4:43 PM, Nathan Sparks <nsparks at jlab.org> wrote:

> Richard,
>
> By default, Scons calculates a md5 checksum of the target file's contents
> to determine if it has changed.
> This takes longer for larger target files, such is the case when debug
> symbols are turned on.
> Also, it will clearly take longer the more plugins you have; you probably
> noticed it after more plugins were added to sim-recon.
>
> David’s pull request which was merged today reduces the average size of
> target files substantially, which should speed up the md5 check.
>
> The behavior of how scons decides to rebuild can be customized but it is
> not clear to me if we should modify it after David’s PR.
> See the documentation on the Decider at
> http://www.scons.org/doc/HTML/scons-user/ch06.html#idm139837621900832
>
> -Nathan
>
> On Mar 23, 2017, at 2:05 PM, Richard Jones <richard.t.jones at uconn.edu>
> wrote:
>
> Natthan,
>
> How can it matter if I have debug on or off, if there is nothing to
> rebuild and nothing to do, just check that everything is up to date and
> return a "nothing to do" message? Up until a few weeks ago, this was very
> fast on my Centos 6 machine. Now it is about 20 minutes and pretty much
> saturates my io to the nfs server over that period. The process is "scons"
> (no compilers or ld or installers are ever invoked) so I cannot see what it
> is doing, except that it is deep in some python loop and reading very
> slowly through every .so and .o file in sim-recon.
>
> -Richard J.
>
> On Thu, Mar 23, 2017 at 10:13 AM, Nathan Sparks <nsparks at jlab.org> wrote:
>
>> Hi Richard,
>>
>> Here are some quick tests using sim-recon commit a5474aa on my (non-Farm)
>> RHEL 7 machine, with a very light load:
>> 1. If I build with debug symbols off (8.5 min. w/ 8 threads) and then run
>> “scons -j8 install DEBUG=0” in the src directory,
>> it takes ~ 15 s for scons to report back that the “install” is up to date.
>>
>> 2. If I rebuild with debug symbols on (19 min. w/ 8 threads) and then run
>> “scons -j8 install” in the src directory,
>> it takes ~ 1.5 min. to output that the installation is up to date.
>>
>> I having been doing my builds with DEBUG=0 for a long time now, so I am
>> used to the quicker builds/checks.
>>
>> Maybe the longer time it takes to check a debug build in combination with
>> a heavy system load can explain your observation.
>>
>> -Nathan
>>
>> > On Mar 23, 2017, at 7:23 AM, Richard Jones <richard.t.jones at uconn.edu>
>> wrote:
>> >
>> > Hello all,
>> >
>> > Did someone check in a change that might have caused this? A few weeks
>> back, if I did "scons install" in sim-recon and there was nothing to do (no
>> updates since last build) it would return with success and report
>> up-to-date within less than 1 minute. Now it takes forever (I hit ctrl-C
>> after 20 minutes) to discover that it has nothing to do.
>> >
>> > -Richard J.
>> > _______________________________________________
>> > Halld-offline mailing list
>> > Halld-offline at jlab.org
>> > https://mailman.jlab.org/mailman/listinfo/halld-offline
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20170323/fed8741e/attachment-0002.html>