[Clas12_software] update

Wed Dec 7 16:42:28 EST 2016

Dear All,

IT security group is still looking into the vulnerability of the Clara monitoring time-series database. As a precaution monitoring is temporarily disabled for users outside of the JLAB.

Regards,
-vardan

> On Dec 6, 2016, at 5:55 PM, Vardan Gyurjyan <gurjyan at jlab.org> wrote:
> 
> Hi Cole,
> I will contact IT security tomorrow. It is ridiculous to block external IP that writes ever 5 sec.  ~100Bytes of data.    How we are going to monitor data acquisition and data processing from collaborating universities? Claraweb is a virtual machine outside of any imaginable firewalls.  The only explanation is that they want to make  sure this requests are not a security bridging attempts. Any ways I will talk to these guys and if it necessary I can make frequency of reporting below the "threshold".
> -Vardan
> 
> Sent from my iPhone
> 
> On Dec 6, 2016, at 5:39 PM, L Smith <lcs1956 at me.com <mailto:lcs1956 at me.com>> wrote:
> 
>> Vardan,
>> 
>> My IP from Norfolk was blocked on Monday by JLAB/CC.  When I made a CCPR I got the first response below.  At first I though it meant my normal mysql connections to clasdb.jlab.org <http://clasdb.jlab.org/> from GEMC, etc.  Later Eric Hacecky sent the 2nd response below.  He claims the query was repeated every 5 seconds, which is enough to trip their threshold.  Apparently your claraweb dashboard was doing something they don’t like.  
>> 
>> I’m not sure what the next step is.  I don’t want my IP banned again but I want to test your stuff.
>> 
>> Thanks, Cole
>> 
>> 
>> **** 1st response ****
>> Your off-site IP was blocked by our IDS/IPS due to pattern matching
>> consistent with sql injection.
>> 
>> In your case we know this is a false positive, and we have measures in
>> place to make sure legitimate traffic from a human is not blocked in
>> this manner.  If you are running a script that makes numerous sql
>> queries in short succession please let me know.
>> 
>> Traffic from the IP you listed exceeded the allowed threshold for sql
>> injection matching which triggered the block.
>> 
>> I'm replying to this CCPR now to let you know the IP is unblocked.
>> 
>> As this is a unique case, I have to go through the traffic to see
>> exactly which sql queries matched over what timeframe and then make
>> necessary adjustments to prevent the same behavior from your IP
>> triggering a block.
>> 
>> I expect to have this in place by close of business today, so any work
>> you may need to do beyond that should not trigger a block as long as
>> the query frequency is not significantly increased.
>> 
>> If you have any additional questions or concerns please let me know.
>> 
>> Regards,
>> Eric Hacecky | JLab Cyber Security
>> ****
>> 
>> **** 2nd response ****
>> 
>> Here's the full log line I extrapolated that from.
>> 
>> //
>> 1480967955.111127	70.161.227.4	49677	129.57.64.89	3000	13	GET
>> claraweb.jlab.org <http://claraweb.jlab.org/>	/api/datasources/proxy/2/query?db=clara&q=SELECT
>> last("exec_time") FROM "rpafecs"."clas12" WHERE "claraSession" =~
>> /^(gurjyan-129\\.57\\.76\\.219%7000_java)$/ AND "service_name" =
>> 'DCTB' AND time > now() - 5m GROUP BY time(500ms)
>> fill(null)&epoch=ms
>> http://claraweb.jlab.org:3000/dashboard/db/pdp-b <http://claraweb.jlab.org:3000/dashboard/db/pdp-b>	1.1	Mozilla/5.0
>> (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like
>> Gecko) Version/9.1.2 Safari/601.7.7	0	16	200	OK	-	-	HTTP::URI_SQLI
>> //
>> 
>> That's the IP you provided.
>> 
>> I guess the good news is, whatever you're doing with clasweb isn't
>> what's blocking the IP.
>> 
>> I figure if you were sharing a connection with gurjyan you would have
>> told me by now.
>> 
>> If that's not the case, the only thing I can think of is Cox assigning
>> the IP to you dynamically after he had it earlier and got it blocked.
>> 
>> Let me know, otherwise you should be in the clear.
>> 
>> To be on the safe side we should run a test tomorrow where you connect
>> to clasweb and I'll watch and make sure nothing is being tripped.
>> 
>> Eric		
>> 
>> ****
>> 
>> 
>> 
>> 
>>> On Dec 6, 2016, at 11:28 AM, Vardan Gyurjyan <gurjyan at jlab.org <mailto:gurjyan at jlab.org>> wrote:
>>> 
>>> This is not a critical update, and is relevant to monitoring.
>>> 
>>> Release notes:
>>> - added pool size reporting for a service. This is an actual core count participating in the processing: Sebastian Mancilla
>>> - fixed bug in the Clara installer, reported by Markov N. This will prevent dangling/leftover processes.
>>> 
>>> Actions:
>>> 
>>> 1) stop or wait until clara reconstruction processes are done.
>>> 
>>> 2) kill dangling Clara processing sessions:
>>> 
>>>> ps -ef | grep java 
>>> will indicate process id’s.
>>>> kill -9 <procesID>
>>> 
>>> 3) update clara installation
>>>> install-claracre-clas -u
>>> 
>>> P.S. at the moment I can see two dangling processes: 
>>> 1) user=celentan on the node=129.57.70.18
>>> 2) user=davita on the node=129.57.70.15
>>> 
>>> Regards,
>>> -vg
>>> 
>>> 
>>> _______________________________________________
>>> Clas12_software mailing list
>>> Clas12_software at jlab.org <mailto:Clas12_software at jlab.org>
>>> https://mailman.jlab.org/mailman/listinfo/clas12_software <https://mailman.jlab.org/mailman/listinfo/clas12_software>
> _______________________________________________
> Clas12_software mailing list
> Clas12_software at jlab.org
> https://mailman.jlab.org/mailman/listinfo/clas12_software

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas12_software/attachments/20161207/5c12614f/attachment-0002.html>