<html>
<head>
</head>
<body text="#000000" bgcolor="#ffffff">
Matt,<br>
<br>
Having seen and worked through a dozen or so SIMD-related issues in
building this software stack on different hardware for the grid, I
have seen no evidence of any "alignment problem", as suspected by
Simon. At any rate, a detailed diagnosis is better than a
suspicion, so here is a detailed diagnosis of the problem you are
seeing, building on stanley.physics.indiana.edu and running on the
K7 nodes of the stan cluster.<br>
<br>
Stanley.physics.indiana.edu is:<br>
<ul>
<li>dual 4-core Intel(R) Xeon(R) CPU X5482 @ 3.20GHz</li>
<li>x86_64 instruction set, 64-bit architecture</li>
<li>running a 32-bit kernel (2.6.18-194.32.1.el5PAE)</li>
<li>supports SIMD extensions: mmx sse sse2 sse3 ssse3 sse4_1 </li>
</ul>
worker nodes on the stan cluster are:<br>
<ul>
<li>dual single-core AMD K7 athlon CPUs @ 1667MHz</li>
<li>i686 instruction set, 32-bit architecture</li>
<li>running a 32-bit kernel (2.6.18-194.32.1.el5)</li>
<li>supports SIMD extensions: mmx sse mmxext</li>
</ul>
In case you would like to verify, I have attached a miniature c++
program that queries the processor for all of the common SIMD
extensions that it supports. You can compile and run this on any
node, and verify the kinds of SIMD extensions that it can execute.<br>
<br>
This Stanley is a bit of an odd-ball: a 64-bit processor running a
32-bit OS. What happens during the build is that the Makefile.SIMD
tries to discover what SIMD support is present in the hardware. You
are building on Stanley, so it queries the processor on Stanley, and
finds it supports sse, sse2, ssse3, and sse4_1. After that, it
builds an executable that exploits all of these features, and that
is what you want -- if you run on Stanley. If you look in the build
logs, you should see the gcc/g++ flags "-mfpmath=sse -msse
-DUSE_SSE2 -msse2" which enables both sse and sse2 instructions.
That code runs fine on stanley, but try to run it on c0-0, and bang,
the parts of the code that try to use sse2 extensions are going to
hit a wall. The xmm registers used by the sse2 extensions are 128
bits, compared with the 64-bit registers used by the sse extensions,
which leads to the segfault you are seeing.<br>
<br>
The immediate solution for you on the stanley cluster is to redo the
build (make clean;make) on one of the worker nodes. Then it will
recognize that sse2 is not supported, and set the flags to
"-mfpmath=sse -msse -mno-sse2", so sse math will be used (consistent
answers) but not the sse2 extensions (no segfaults). This code will
run both on the K7 nodes and on the head node, and will give answers
that are consistent with running full Simon-supercharged code on a
64-bit node, but without the super-charged performance.<br>
<br>
For the future, we should have a run-time-startup check in our
applications that verifies that the options used during the build
are supported by the cpu running the code. This is easy to do, and
for code that uses DANA, it is now a part of the Init() method of
the DApplication class -- I added it last week. It would be trivial
to copy that code into the main() of non-DANA apps like mcsmear and
hdgeant. I would support that, but will hold back on checking in
more SIMD-related changes until some of this dust settles and we
know that things are under control.<br>
<br>
-Richard J.<br>
<br>
<br>
<br>
<br>
<br>
On 3/2/2011 6:08 PM, Simon Taylor wrote:
<blockquote type="cite" cite="mid:4D6ECE04.9050803@jlab.org">
<pre wrap="">Hi.
Our suspicion is that there is an alignment problem on 32-bit systems
with regard to the SIMD instructions; Dave looked into this some time
ago and it is not clear to us how to fix it.
I've checked in a change to Makefile.SIMD that changes the default from
"SIMD on" to "SIMD off". To get the SIMD instructions, one would
now need to do "make ENABLE_SIMD=yes".
Simon
Matthew Shepherd wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi all,
It seems that the BMS system doesn't properly understand our SIMD capabilities on the machines here at Indiana. If we do a default build, then we get a segmentation fault at the first DVector2 operation. If we build with DISABLE_SIMD=1 then this segfault is avoided.
This seems to point to two causes:
(1) there is a bug in the SIMD implementation of DVector2
(2) our machines are not capable of handling current SIMD code
(1) seems unlikely since other people are using the code. Assuming it is (2), how do we properly diagnose and fix it?
-Matt
_______________________________________________
Halld-offline mailing list
<a href="mailto:Halld-offline@jlab.org">Halld-offline@jlab.org</a>
<a href="https://mailman.jlab.org/mailman/listinfo/halld-offline">https://mailman.jlab.org/mailman/listinfo/halld-offline</a>
</pre>
</blockquote>
<pre wrap="">
_______________________________________________
Halld-offline mailing list
<a href="mailto:Halld-offline@jlab.org">Halld-offline@jlab.org</a>
<a href="https://mailman.jlab.org/mailman/listinfo/halld-offline">https://mailman.jlab.org/mailman/listinfo/halld-offline</a>
</pre>
</blockquote>
<br>
</body>
</html>