J. Kissel, K. Kizumi, S. Karki

While trying to commit our results from last night to the CalSVN, we discovered that every folder was complaining that the it was not a working copy of svn, e.g.:

jeffrey.kissel@opsws5:/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Scripts$ svn st
svn: warning: '.' is not a working copy

Totally confused, we went to the trunk of the repo, and it said messages about discrepant svn client versions:

/ligo/svncommon/CalSVN/aligocalibration$ svn st
svn: The path '.' appears to be part of a Subversion 1.7 or greater
working copy.  Please upgrade your Subversion client to use this
working copy.


In order to make progress on the work stations (i.e. the h1boot server) Kiwamu made a back of the repo into 
/ligo/svncommon/CalSVN/aligocalibration_backup_20150530/
and made a fresh checkout of the repo, in the normal path:
/ligo/svncommon/CalSVN/aligocalibration
which regained its normal functionality.

He then started digging around the log history to figure out what happened and found a bunch of commits from the LHO PCAL team yesterday evening... and then Sudarshan joined us for a happy (??) Saturday discovery.

We've diagnosed the problem (a totally innocent user just doing what the computer told him to do -- he should not be blamed for any of this!!):
Sudarshan was at the End Station trying to check in pictures of PCal camera on (new? old?) CDS Laptop called cdsmbp0. He tried to do an svn st just to check on things, and an svn update before he committed (very good!), but he got the following error message:
[controls@cdsmbp0 Measurements]$ svn st
svn: E155036: Please see the 'svn upgrade' command
svn: E155036 Working copy '/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements' is too old (format 10, created by Subversion 1.6)
[controls@cdsmbp0 Measurements]$ svn update
svn: E155036: Please see the 'svn upgrade' command
svn: E155036 Working copy '/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements' is too old (format 10, created by Subversion 1.6)

He obediently then followed the instructions:
[controls@cdsmbp0 Measurements]$ svn upgrade
svn: E155019: Can't upgrade '/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements' as it is not a pre-1.7 working copy root, the  root is '/ligo/svncommon/CalSVN/aligocalibration'
[controls@cdsmbp0 Measurements]$ cd ../../../
[controls@cdsmbp0 Runs]$ cd ../../
[controls@cdsmbp0 aligocalibration]$ ls
branches tages trunk
[controls@cdsmbp0 aligocalibration]$ svn upgrade
Upgraded '.'
Upgraded 'trunk'
Upgraded 'trunk/Runs/'
#etc
#etc
#HERE'S A CRAZY BIT
Upgraded 'trunk/Runs/PreER7/H1/Measurements/FreeSwingMich/2015-05-27'
Upgraded 'trunk/Runs/PreER7/H1/Measurements/FreeSwingMich/2015-05-28'
nfs server cdsfs0:/ligo: not responding                               # <<< what the heck??
nfs server cdsfs0:/ligo: is alive again
Upgraded 'trunk/Runs/PreER7/H1/Measurements/DARMASDs/'                # <<< OK... moving on... ??
Upgraded 'trunk/Runs/PreER7/H1/Measurements/ElectronicsMeasurements/'
#etc
#etc
Upgraded 'branches'
Upgraded 'tags'
[controls@cdsmbp0 aligocalibration]$

This upgraded the hidden .svn configuration file in the entire repo in a way that is incompatible with any version of the svn client prior to 1.7. YUCK!!

It turns out that every computer does not source the svn client binary from the h1boot server as I would have imagined -- so this laptop was using its own copy of the svn binary /usr/bin/svn, which was of a later version than what is normally used on the work stations:
work stations: svn, version 1.6.17 (r1128011) compiled Aug 13 2014, 20:41:52
this laptop: svn, version 1.7.10 (r1485443) compiled Aug 13 2013, 15:31:22

What's worse, is that this "upgrade" to svn version 1.7.10 caused the entire h1boot server's copy repo to fail on the work stations as shown above immediately degrading the whole repo to just a normal unrepoed directory structure and it's not at all backward compatible. Even still more confusing -- svn version 1.6.17 doesn't even have an option to svn upgrade so we didn't believe (at first) what Sudarshan did was even possible.

Thankfully
- Sudarshan left the terminal open on this laptop so we were able to document everything that happened in great detail from this laptop's terminal session
- Kiwmau has made the back up mentioned above, 
- *and* Sudarshan ran an svn st after the upgrade on this laptop, so we know everything that wasn't committed of was modified. (see attached)
So nothing should be lost, we know the status of everything, and we can work to merge the backup back into the repo and move with out too much guess work.

However -- the SysAdmins (don't worry -- this wasn't your fault either!! This is a *terrible* backward compatibility issue with the SVN clients. I blame them!!):
 We should make sure all computers on site running svn cleint 1.6.17. IF and when we want to upgrade to svn client 1.7 or higher, we need to be absolutely sure that we do it to all computers at once.
OR we should demand that all computers are using a client that is hosted on the h1boot server, such that no computer is using an individual copy of an svn client.
 
#scaryscaryscary
#yakshaving