<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Dear All,</div><div class=""><br class=""></div>IT security group is still looking into the vulnerability of the Clara monitoring time-series database. As a precaution monitoring is temporarily disabled for users outside of the JLAB.<div class=""><br class=""></div><div class="">Regards,</div><div class="">-vardan<br class=""><div class=""> <br class=""><div><blockquote type="cite" class=""><div class="">On Dec 6, 2016, at 5:55 PM, Vardan Gyurjyan <<a href="mailto:gurjyan@jlab.org" class="">gurjyan@jlab.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="content-type" content="text/html; charset=utf-8" class=""><div dir="auto" class=""><div class="">Hi Cole,</div><div class="">I will contact IT security tomorrow. It is ridiculous to block external IP that writes ever 5 sec. ~100Bytes of data. How we are going to monitor data acquisition and data processing from collaborating universities? Claraweb is a virtual machine outside of any imaginable firewalls. The only explanation is that they want to make sure this requests are not a security bridging attempts. Any ways I will talk to these guys and if it necessary I can make frequency of reporting below the "threshold".</div><div class="">-Vardan<br class=""><br class="">Sent from my iPhone</div><div class=""><br class="">On Dec 6, 2016, at 5:39 PM, L Smith <<a href="mailto:lcs1956@me.com" class="">lcs1956@me.com</a>> wrote:<br class=""><br class=""></div><blockquote type="cite" class=""><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div class="">Vardan,</div><div class=""><br class=""></div><div class="">My IP from Norfolk was blocked on Monday by JLAB/CC. When I made a CCPR I got the first response below. At first I though it meant my normal mysql connections to <a href="http://clasdb.jlab.org/" class="">clasdb.jlab.org</a> from GEMC, etc. Later Eric Hacecky sent the 2nd response below. He claims the query was repeated every 5 seconds, which is enough to trip their threshold. Apparently your claraweb dashboard was doing something they don’t like. </div><div class=""><br class=""></div><div class="">I’m not sure what the next step is. I don’t want my IP banned again but I want to test your stuff.</div><div class=""><br class=""></div><div class="">Thanks, Cole</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">**** 1st response ****</div><div class="">Your off-site IP was blocked by our IDS/IPS due to pattern matching<br class="">consistent with sql injection.<br class=""><br class="">In your case we know this is a false positive, and we have measures in<br class="">place to make sure legitimate traffic from a human is not blocked in<br class="">this manner. If you are running a script that makes numerous sql<br class="">queries in short succession please let me know.<br class=""><br class="">Traffic from the IP you listed exceeded the allowed threshold for sql<br class="">injection matching which triggered the block.<br class=""><br class="">I'm replying to this CCPR now to let you know the IP is unblocked.<br class=""><br class="">As this is a unique case, I have to go through the traffic to see<br class="">exactly which sql queries matched over what timeframe and then make<br class="">necessary adjustments to prevent the same behavior from your IP<br class="">triggering a block.<br class=""><br class="">I expect to have this in place by close of business today, so any work<br class="">you may need to do beyond that should not trigger a block as long as<br class="">the query frequency is not significantly increased.<br class=""><br class="">If you have any additional questions or concerns please let me know.<br class=""><br class="">Regards,<br class="">Eric Hacecky | JLab Cyber Security</div><div class="">****</div><div class=""><br class=""></div><div class="">**** 2nd response ****</div><div class=""><br class=""></div><div class="">Here's the full log line I extrapolated that from.<br class=""><br class="">//<br class="">1480967955.111127<span class="Apple-tab-span" style="white-space: pre;"> </span>70.161.227.4<span class="Apple-tab-span" style="white-space: pre;"> </span>49677<span class="Apple-tab-span" style="white-space: pre;"> </span>129.57.64.89<span class="Apple-tab-span" style="white-space: pre;"> </span>3000<span class="Apple-tab-span" style="white-space: pre;"> </span>13<span class="Apple-tab-span" style="white-space: pre;"> </span>GET<br class=""><a href="http://claraweb.jlab.org/" class="">claraweb.jlab.org</a><span class="Apple-tab-span" style="white-space: pre;"> </span>/api/datasources/proxy/2/query?db=clara&q=SELECT<br class="">last("exec_time") FROM "rpafecs"."clas12" WHERE "claraSession" =~<br class="">/^(gurjyan-129\\.57\\.76\\.219%7000_java)$/ AND "service_name" =<br class="">'DCTB' AND time > now() - 5m GROUP BY time(500ms)<br class="">fill(null)&epoch=ms<br class=""><a href="http://claraweb.jlab.org:3000/dashboard/db/pdp-b" class="">http://claraweb.jlab.org:3000/dashboard/db/pdp-b</a><span class="Apple-tab-span" style="white-space: pre;"> </span>1.1<span class="Apple-tab-span" style="white-space: pre;"> </span>Mozilla/5.0<br class="">(Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like<br class="">Gecko) Version/9.1.2 Safari/601.7.7<span class="Apple-tab-span" style="white-space: pre;"> </span>0<span class="Apple-tab-span" style="white-space: pre;"> </span>16<span class="Apple-tab-span" style="white-space: pre;"> </span>200<span class="Apple-tab-span" style="white-space: pre;"> </span>OK<span class="Apple-tab-span" style="white-space: pre;"> </span>-<span class="Apple-tab-span" style="white-space: pre;"> </span>-<span class="Apple-tab-span" style="white-space: pre;"> </span>HTTP::URI_SQLI<br class="">//<br class=""><br class="">That's the IP you provided.<br class=""><br class="">I guess the good news is, whatever you're doing with clasweb isn't<br class="">what's blocking the IP.<br class=""><br class="">I figure if you were sharing a connection with gurjyan you would have<br class="">told me by now.<br class=""><br class="">If that's not the case, the only thing I can think of is Cox assigning<br class="">the IP to you dynamically after he had it earlier and got it blocked.<br class=""><br class="">Let me know, otherwise you should be in the clear.<br class=""><br class="">To be on the safe side we should run a test tomorrow where you connect<br class="">to clasweb and I'll watch and make sure nothing is being tripped.<br class=""><br class="">Eric<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span></div><div class=""><br class=""></div><div class="">****</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Dec 6, 2016, at 11:28 AM, Vardan Gyurjyan <<a href="mailto:gurjyan@jlab.org" class="">gurjyan@jlab.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">This is not a critical update, and is relevant to monitoring.<br class=""><br class="">Release notes:<br class="">- added pool size reporting for a service. This is an actual core count participating in the processing: Sebastian Mancilla<br class="">- fixed bug in the Clara installer, reported by Markov N. This will prevent dangling/leftover processes.<br class=""><br class="">Actions:<br class=""><br class="">1) stop or wait until clara reconstruction processes are done.<br class=""><br class="">2) kill dangling Clara processing sessions:<br class=""><br class=""><blockquote type="cite" class="">ps -ef | grep java <br class=""></blockquote>will indicate process id’s.<br class=""><blockquote type="cite" class="">kill -9 <procesID><br class=""></blockquote><br class="">3) update clara installation<br class=""><blockquote type="cite" class="">install-claracre-clas -u<br class=""></blockquote><br class="">P.S. at the moment I can see two dangling processes: <br class="">1) user=celentan on the node=129.57.70.18<br class="">2) user=davita on the node=129.57.70.15<br class=""><br class="">Regards,<br class="">-vg<br class=""><br class=""><br class="">_______________________________________________<br class="">Clas12_software mailing list<br class=""><a href="mailto:Clas12_software@jlab.org" class="">Clas12_software@jlab.org</a><br class=""><a href="https://mailman.jlab.org/mailman/listinfo/clas12_software" class="">https://mailman.jlab.org/mailman/listinfo/clas12_software</a></div></div></blockquote></div><br class=""></div></blockquote></div>_______________________________________________<br class="">Clas12_software mailing list<br class=""><a href="mailto:Clas12_software@jlab.org" class="">Clas12_software@jlab.org</a><br class="">https://mailman.jlab.org/mailman/listinfo/clas12_software</div></blockquote></div><br class=""></div></div></body></html>