[Halld-controls] Fw: MAJOR changes to /site and /apps on central fileserver, Tuesday Nov 17 -- MUST READ !

[Hovanes Egiyan]

________________________________
From: Paul Letta <letta at jlab.org>
Sent: Wednesday, November 11, 2020 7:46 AM
To: Ole Hansen <ole at jlab.org>; Brad Sawatzky <brads at jlab.org>; Stephen Wood <saw at jlab.org>; Sergey Boyarinov <boiarino at jlab.org>; Sergey Furletov <furletov at jlab.org>; Hovanes Egiyan <hovanes at jlab.org>; Nathan Baltzell <baltzell at jlab.org>; David Abbott <abbottd at jlab.org>; Anthony Cuffe <cuffe at jlab.org>; Adam Carpenter <adamc at jlab.org>; Erik Werlau <werlau at jlab.org>; Kari Heffner <heffner at jlab.org>; Bobby Lawrence <robertl at jlab.org>; David Sheppard <sheppard at jlab.org>; unixadm at jlab.org <unixadm at jlab.org>; secops at jlab.org <secops at jlab.org>
Subject: MAJOR changes to /site and /apps on central fileserver, Tuesday Nov 17 -- MUST READ !


We will be making major changes to the file servers listed below on Tuesday, Nov 17, 2020



Jlabsite:/site

Jlabapps:/appsroot



If any of your machines mount these areas.. you need to act on this.



Currently jlabsite and jlabapps are actually aliases that point to the same NetApp, cfs1a.jlab.org.

We are moving these areas to new virtual filers on new NetApp hardware.  The IP addresses that both of these resolve to now,  will  change and will be different from each other.



So – the IP addresses that jlabsite and jlabapps resolve to right now, will be changing.   This has major implications for MOST linux machines at JLAB.



We have published these to all JLAB machines via the automounter maps for many years.. therefore the number and location of machines that actually use them is impossible for us to know.

(mainly because many other exports share the same NetApp right now).



Here is the general plan:

·         Tuesday, 11/17 at 5PM – change exports of jlabsite:/site and jlabapps:/appsroot to Read Only for all hosts.   This isn’t that big a change, as there are only handful of machines that have r/w access to it.

·         Do a final incremental copy – this should take less than an hour.

·         Change DNS so that jlabsite and jlabapps now point to their new IPs.

·         Tuesday, 11/17 at about 6PM – start rebooting all Linux machines that we have controls over – including desktops.

·         Tuesday, 11/17 at about 8PM – remove even the ReadOnly exports from the old filer’s IP address for jlabsite:/site and jlabapps:/appsroot.



It’s very important that you plan to reboot Linux hosts that the IT division does not patch/reboot monthly..  If you reboot after 6PM but BEFORE 8PM.. they will likely go down and reboot just fine.

If you wait until the areas are no longer exported from the old IP… your machines will likely hang trying to unmount these areas, as they will be stale.  In this case, usually a physical intervention is required (hard power cycle, or  Alt-SysRq-B).  This is the situation that we are trying to avoid.



For the Windows machines – windows servers (MIS) and desktops will cache the DNS entries of the old IP address – a reboot (After 6PM) will clear this.



Let me know of any questions.

Thanks,

Paul Letta

letta at jlab.org<mailto:letta at jlab.org>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-controls/attachments/20201111/41d0838f/attachment.html>