[Halld-offline] [EXTERNAL] CCDB interruption today

[Mark Ito]

People,

Marty has complete the restore from backup of the CCDB. Recall that any 
constants written between 3:45 am and a bit ago have been lost. This 
restoration appears to have propagated correctly to all replicas (e.g., 
hallddb-farm). Please let me know if you see problems using the CCDB 
from the MySQL servers.

One note. There were a few active connections to the server when the 
restore was attempted. Those connections (about 20 of them) had to be 
terminated by hand. Most of these jobs probably went through without any 
bad effects, but some of them may have crashed and will have to be 
restarted.

   -- Mark

On 2/19/2020 3:09 PM, Mark Ito wrote:
> People,
>
> To make sure we restore all of the data from this incident we are 
> rolling back to a backup of the CCDB database done at 3:45 am this 
> morning. Please refrain from writing new constants to the CCDB until 
> this is done (on the master server, hallddb.jlab.org). Anything 
> written now will be lost. Marty Wise of the computer center is doing 
> the restore and should be done in an hour or so.
>
>   -- Mark
>
> On 2/19/2020 2:44 PM, Sean Dobbs wrote:
>> Hi Offliners,
>>
>> Due to the temporary disappearance of a CCDB table, jobs
>> reconstructing raw data that accessed the master CCDB would have
>> crashed from about 11 am - 2:30 pm today.  This should be fixed now.
>> Please let me know if you see any other problems.
>>
>> Cheers,
>> Sean
>> _______________________________________________
>> Halld-offline mailing list
>> Halld-offline at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/halld-offline
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline