[Clas_online] clasweb upgrade - effects logbook and run database

[boiarino at jlab.org]

>From Marty Wise:

The Computer Center has been planning the upgrade of clasweb for a while
now, and I have been expecting to work on it in earnest after the first of
the year. Due to increasingly frequent problems with the logbook
application, Sergey is anxious to get it done sooner, and the holiday break
represents a window of opportunity that is worth considering. My original
plan was to advertise the information below after we returned from the
holiday and set up the new server in parallel and work on setting up DB
access for it with the original system still online. Then, we could test
things, take care of most of the cleanup and small problems, and ultimately
schedule the cutover for the next convenient opportunity. Please review the
info below and let me know if you want me to proceed.  If so, I’ll move
forward over the holidays with a couple of important notes


1. This will be messy and some stuff will break. I will work with Sergey to
make sure the logbook and data browser apps work (needed in the control
room) before we return.
2. Some apps will take some time to identify and fix. Some, rarely used
stuff may not get fixed, because it doesn’t get reported.

I have the new replacement for clasweb built. It has been in service for
several months now, hosting DocDB, a MantisBT problem tracking system and a
wiki for TPE. I recently got Greg Riccardi’s tomcat-based logbook app
apparently working on the new system. It’s difficult to test things very
well since many applications require access to databases on clasdb and other
systems, and these ACLs aren’t set up, but it appears that the logbook will
work on the new system as desired.

This is a major upgrade of all the web components and the underlying OS and
so will very likely break some (maybe many?) applications. For the most
part, the problems likely to be encountered should be easy to fix, but will
take a little time to identify and track down. While there is the
possibility of numerous small issues, there are a couple of configuration
changes that  are very likely to cause some problems that will need fixing.
I will try to get things corrected, but there will probably be some cases
where the software owner will need to be involved. This is information that
I’d like to be able to let developers/maintainers know about. I’ve been
assuming I’d be able to advertise this stuff for a month or so before the
upgrade to allow people to prepare


I hope you all have a nice holiday.

Marty



Config Changes That Are Likely To Cause Problems:
====================================================================
1. CUE Web servers  no longer mount all of group by default. As a security
measure, we mount only the portions of group that are needed. Applications
that reference areas of group not initially mounted will have problems. Once
these are identified, the necessary areas can be mounted as required. While
developing the initial configuration for the new server, I identified a
number of areas that were accessed and have already mounted them. Currently,
claswebnew mounts:
  * /group/clas/www/clasweb
  * /group/clas/tools
  * /group/clas/parms (READ-ONLY)
  * /group/clas/clas_cvs
  * /group/primex/CVS_REPOSITORY
  * /group/clas/builds
  * /group/clas/12gev/svnroot
  * /group/clon (READ-ONLY)
  * /group/12gev_phys/svnroot
  * /group/halld/Repositories/svnroot

2. The apache web server now run as the user “apache” by default rather than
“httpd” as in the past. This is a real pain, and is a change to keep up with
the defaults used on RedHat Linux. I’ve converted nearly 70 other web
servers so far, and this is usually easy enough to fix
 just a matter of
finding and changing file ownerships. In a pinch, we may be able to retain
the old configuration, but I’d rather not if possible.

3. *** IMPORTANT *** There is a parameter in PHP that controls variables
that refer to data from the web server session (username, remote host, all
get and post variables, etc). The parameter is known as “REGISTER_GLOBALS”.
Historically, it’s value was true, but it changed to false several years ago
due to serious security concerns about the old mode of operation. The new
server will have this value set to false. If you have very old applications
that access these data in the old way, they will break, requiring code
changes to fix.

4. Database Applications -- Ordinarily, when a DB application is configured
or created, some account is created on the DB server to allow the
application to connect and do what it needs to do. For MySQL and other
databases, these accounts also can be restricted by host. In this way, an
account can be allowed to connect only from specific systems. If
applications on clasweb need to connect to various databases in this
fashion, and these connection accounts are so restricted, these apps running
the new system will have problems until the connection accounts are updated
to reflect the new server name. The old server connects as the system
"claspc28", whereas the new system will actually connect directly as
"clasweb", once it goes into production.

Upgrade/Migration Process:
====================================================================

Merge the newclasweb content area into clasweb’s
The content area (including html and cgi directories) is in /group. Clasweb
and newclasweb currently have separate directories in group. Newclasweb’s
content includes the several new apps that I’ve been asked to set up. The
first order of business will be to merge this into the “main” content area
for clasweb (/group/clas/www/clasweb) . Once this is complete, the two
servers should “look” identical to users with a couple of exceptions:

1. Some apps that use DB connections may not work on the new server. If the
application and DB server are set up to only allow connections from the
current server (“claspc28.jlab.org”), then the applications won’t be able to
connect to their databases and will not work. Even when the transition is
finished, these connections will need to be fixed. The new system will
ultimately be called “clasweb” (not “claspc28”), and so connections that are
permitted based on the claspc28 name will need to be updated to reflect the
new name. I can watch for signs of these problems in logs and
reverse-engineer where they’re connecting to, and work on updating the
access entries. I can probably take care of anything that connects to
clasdb, or any of several central MySQL servers. For those that connect to
DB servers that you guys (or others) admin, I won’t be able to fix them. I
will, however, collect information about the errors from the logs and try to
contact the relevant system owner to see about getting updates made.

2. If applications try to access areas of group that aren’t mounted, they
won’t work. I can watch the server logs for signs of these problems and
mount the required directories.

3. While testing with the server named claswebnew, you have to be watchful
for links that explicitly point to clasweb -- possibly moving you un-noticed
back onto the production server.

Test As Many Applications as Possible

Note that the logs mentioned above will only show errors if the various
applications are accessed. Over the holiday, it’s unlikely that there will
be much access, so much of the above won’t become apparent until after the
system goes into production and everyone returns.

Shut down the httpd service on the old clasweb server (claspc28)

1. The httpd service itself on claspc28 will be stopped and disabled from
starting automatically at boot
2. Additionally, the second IP address used for clasweb will be
un-configured on claspc28 and disabled from restarting at boot
3. The current DNS entry for clasweb will be removed

Rename the system currently called “claswebnew” to clasweb

This requires numerous small changes in config files, DNS, filer exports,
etc. This process is well known and used for numerous server migrations in
the past. In this case, I will use the IP address previously assigned to
clasweb as the IP for the new clasweb system. This will allow offsite users
to have access immediately (without the usual delays for DNS propagation),
and will use the existing firewall penetrations, etc.

Test, Test, Test


I’ll try to have the migration described above complete at least a couple of
days before our return. This will allow some time for anyone interested to
test things out and let me know about any problems. There are several
“fixes” I can apply without testing
 For instance:
• I plan to run a find on the clasweb content area to locate files that are
specifically owned or grouped to “httpd” and change these to “apache”.
• I can also find all the .htaccess files and look for anything that might
be a problem
• I have already made a list of wiki installations on clasweb, I can check
these for the DBs they connect to and try to fix any connectivity issues.
• I can examine the apache access and error logs from the past to look for
commonly access applications and check for config that might be a problem on
the new server.

Upon our Return

Upon our return after the first of the year, I will plan to be available for
the first couple of days at least to address any problems that are
identified on the new server. I expect the first day to be pretty busy
fixing things, but this should trail off quickly.