Version Control - UNIX configuration files

Last Revised: 29 Jan. 2003

Companion Documents

CVS Conventions and Policy.
Information highly specific to our project.

Introduction

There are two basic methods for putting exist unix config files under CVS, the traditional CVS import/checkout method, described below for Oracle, and the non-invasive method, describe below for /etc.
Note that the two methods are basically identical when importing files into CVS, they differ only on how to convert the existing non-CVS are into a proper CVS checkout. The traditional method simply does a new CVS checkout and deletes the old non-CVS area. The non-invasive method grafts the proper CVS directories into the existing non-CVS area.
I strongly suggest using the non-invasive method, for /etc, for Oracle, in short, for most sysadmin type things. If there is little or no risk to removing and replacing the files temporarily, then the traditional method should be ok. But if the files should stay in place with the guaranteed exact same unix permissions throughout the process, then you are better off with the non-invasive method - it is lower risk.

Putting Oracle 8.1.7 under CVS

Various Oracle configuration files are both critical to the proper operation, and are occasionally tweaked and modified and just plain fooled with by DBAs and Sysadmins. Clearly, we want these files under some form of version control...
After consideration of the standard Oracle 8.1.7 directory structure, we chose to root our Oracle CVS project at $ORACLE_BASE (which is frequently /ora8/m01/app/oracle).
Before we start, let's set the group sticky bit on all the directories under $ORACLE_BASE. That way, if the group of a directory is "dba" and I create a file in there, the group of the file will also be "dba", even if my default group is "users" or "staff" or "atp" something like that:
$ sudo find $ORACLE_BASE -type d -exec chmod g+s {} \;

Now let's import some of the config files into CVS. We don't really want to check all 2.3 GB of the Oracle stuff into CVS, so we need to be precise in just how we do the imports:
Note that below I chose to use vendor branch 1.1.25, but you probably should just leave out the -b option altogether and use the default 1.1.1 vendor branch.

$ cd $ORACLE_BASE/product/8.1.7/network/admin/ $ cvs import -b 1.1.25 oracle8/product/8.1.7/network/admin Oracle ora8-config-2001-09-12 $ cd $ORACLE_BASE/product/8.1.7/hs/admin $ cvs import -b 1.1.25 oracle8/product/8.1.7/hs/admin Oracle ora8-config-2001-09-12 $ cd $ORACLE_BASE/admin/ora8/pfile/ $ cvs import -b 1.1.25 oracle8/admin/ora8/pfile/ Oracle ora8-config-2001-09-12

Since we'd already been running our Development copy of Oracle for some time, my cvs commit comment was something like, "Importing Oracle 8.1.7 config files. Note that we have already customized SOME of these files." Ideally, we would have put these files under CVS immediately after installing Oracle, and I recomend that your do so in the future if you have the opportunity!
Also, some of the config files come from Oracle with CVS Header or Id strings in the file, so if you want to preserver those (probably a good idea), go and delete the first "$" from the line now. E.g., in this directory:
$ORACLE_BASE/product/8.1.7/network/admin/
Edit these lines in these files:
namesdrp.sql:Rem Header: namesdrp.sql 26-sep-97.13:45:53 eminer Exp $
namesini.sql:Rem Header: namesini.sql 09-may-97.13:50:14 rkasamse Exp $
namesupg.sql:rem Header: namesupg.sql 09-may-97.13:50:14 rkasamse Exp $

$ cvs commit -m "Tweaked Header$ from Oracle so will not be changed by CVS." namesdrp.sql namesini.sql namesupg.sql
Other Oracle files you should probably also put under CVS:

$ORACLE_BASE/product/8.1.7/assistants/dbca/install/
All the *.sh scripts used to create your datbase. Depending on what you chose in dbassist, yours may be in a slightly different location. But you will often want to edit these scripts before using them to create your database, and may want them for reference later, so put them under CVS.

Ok, now we have a copy of all the config files we want in CVS, but we're still using the old files to actually run our Oracle instance. We need to go replace those files with a working copy checked out from CVS.

Using the traditional CVS import/checkout method:
Note that we could, for example, magically turn the existing $ORACLE_BASE/product/8.1.7/network/admin/ directory into a CVS checkout by manually creating the CVS subdirectory and its appropriate Entries, Repository, and Root files. But we won't do it that way. Instead, we will use the cvs checkout command.
$ cd $ORACLE_BASE/product/8.1.7/network/ $ mv admin admin-old $ cvs co -d admin oracle8/product/8.1.7/network/admin

Now sometimes CVS changes the file permissions, so let's compare the permissions on the files in the new admin/ vs. those in admin-old/. (You can use a command like "find admin -ls" to help you do this.) In our case, the following two commands were sufficient to tweak the permissions in the new admin working copy back to what we wanted:

$ find admin -type d -exec chmod o+rx {} \; $ chmod 644 admin/samples/*.ora

Finally, I did the cvs checkout as my, but before we started meddling all these files were owned by the user oracle, not by me. So let's change them back that way:

$ sudo chown -R oracle admin

We're done putting $ORACLE_BASE/product/8.1.7/network/admin/ under CVS control. Now we just need to do the exact same steps for each all the other separate directories we imported into CVS. E.g.:
$ cd $ORACLE_BASE/admin/ora8/ $ mv pfile pfile-old $ cvs co -d pfile oracle8/admin/ora8/pfile $ chmod 775 pfile $ sudo chown -R oracle pfile $ cd $ORACLE_BASE/product/8.1.7/hs/ $ mv admin admin-old $ cvs co -d admin oracle8/product/8.1.7/hs/admin $ chmod 775 admin $ sudo chown -R oracle admin

Alternately, using the non-invasive method:

First read more details about the non-invasive method below under Putting /etc under CVS. Then come back here, and as the unix user oracle, basically do these steps:
$ export ORA_CVS_TMP=/tmp/ora-cvs
$ mkdir $ORA_CVS_TMP
$ chgrp dba $ORA_CVS_TMP  ;# oracle must be in dba group
$ chmod g+s $ORA_CVS_TMP
$ cvs co -d $ORA_CVS_TMP ora8-koudelka
$ find $ORA_CVS_TMP -name CVS -prune -o -type f -print | xargs rm

$ ./cp-cvs-etc.tcl $ORA_CVS_TMP/product/8.1.7/network/admin/            $ORACLE_BASE/product/8.1.7/network/admin/
$ ./cp-cvs-etc.tcl $ORA_CVS_TMP/product/8.1.7/hs/admin/                 $ORACLE_BASE/product/8.1.7/hs/admin/
$ ./cp-cvs-etc.tcl $ORA_CVS_TMP/admin/ora8/pfile/                       $ORACLE_BASE/admin/ora8/pfile/
$ ./cp-cvs-etc.tcl $ORA_CVS_TMP/product/8.1.7/assistants/dbca/install/  $ORACLE_BASE/product/8.1.7/assistants/dbca/install/

Putting `/etc` under CVS

Using the non-invasive method:

UNIX systems typically store lots and lots of configuration files and other stuff in /etc. Many of the most important files in /etc are actually symlinks, so it would be very nice if our version control tool could handle symlinks. Handling other file properties like permissions, time stamps, etc. would be awfully nice too. Unfortunately, CVS (v. 1.11) currently supports none of these. But, just putting the regular files in /etc into CVS is still very useful - so we'll do that.

Do cvs import as root or not?

Should you import everything into cvs as root, or as an unprivileged unix user? There are two schools of thought here. By putting certain sensitive files like /etc/shadow into CVS, you have introduced another security concern to worry about. You must make sure you set the permissions of those sensitive files in the CVS repository appropriately.
Alternately, if you do the cvs import without sudo (as a non-root user), then you will import only those files which an ordinary user (e.g., you) are allowed to read. This simplifies your security concerns, but, there are various files in /etc/ readable only be root which would still be very nice to keep under version control (e.g., lilo.conf, perhaps ssh/ssh_host_*key).
The first time I did this (on a Solaris 8 machine), I did the cvs import using my own unprivileged unix account. The second time (on a Debian GNU/Linux 3.0 machine), I did the import with as root, with sudo. Both ways work.

Prepare to do the cvs import:

Before we start, let's check what files in /etc have been modified in the last 24 hours:
$ sudo find /etc -mtime -1 -ls

Now let's import the files. However, sometimes when CVS can't read a file, it just stops. Needless to say, this is not what we want - we want it to keep going with the import. Therefore, below we manually exclude each of these problematic files from the import with the -I option. You will probably have a different list of files - just figure it out by trial and error with the import command. Do a cvs import into some test repository. The list of files below is from Solaris, without using sudo. When I did this on Debian 3.0, using sudo, I did not need to exclude any files with -I. Test it yourself first.
Now, cvs -k options: If you're in a hurry and want to make sure CVS doesn't hose your files, just use -kb for everything. If you want to be more easily merge new versions of files in later, use -ko for text files, and only use -kb for files that are really binary. For some files, you will probably want no -k flags at all, but you should be able to remove the -ko flag later on a file by file basis, so I suggest just doing all text files with -ko for now.
TODO: After everything else here is done, I can't seem to remove the -k sticky options with "cvs update -A FILE". That command should work but does not. Fix how? --atp@piskorski.com, 2002/12/05 16:28 EST
TODO: Give advice on whether or not to use the -b 1.1.27 --atp@piskorski.com, 2002/12/05 12:31 EST

Do the import, using -kb for everything:

$ cd /etc/ $ cvs import -b 1.1.27 -kb -I shadow -I .pwd.lock -I initpipe -I oshadow -I .name_service_door -m "Importing everything in /etc/ as of 2001-09-12, whatever the heck it is." unix/etc Sun slash-etc-2001-09-12

Note that I used the -kb flag above, to import everything as binary. Since this is an already working unix machine, this is safest, as it guarantees that tag expansion will not change anything in the file. TODO:atp

Or, instead do the import in two steps, using -ko and -kb:

We need to separate the text from the binary files. So make a tarball of everyting in /etc, untar it to a work area, and make a list of which files are binaries:
$ mkdir ~atp/e
$ sudo tar cf ~atp/e/etc-backup.tar -C /etc /etc
$ cd ~atp/e/
$ sudo tar xpf etc-backup.tar

$ cd ~atp/e/etc
$ sudo find . -type f -print | sudo xargs file > ~atp/e/etc-all-files
$ cat ~atp/e/etc-all-files | sed -e '/text/d'  > ~atp/e/etc-not-text-files
On Debian 3.0, my list of binary files in ~atp/e/etc-not-text-files was:
./terminfo/ ./mc/mc.lib ./X11/rstart/rstartd.real ./X11/app-defaults/XF86Cfg ./ssh/ssh_host_key ./console/boottime.kmap.gz ./ioctl.save ./.pwd.lock ./mime-magic.dat ./ld.so.cache ./emacs21/site-start.d/42hyperlatex.elc ./emacs21/site-start.d/00debian-vars.elc ./xemacs21/site-start.d/42hyperlatex.elc ./xemacs21/site-start.d/00debian-vars.elc
Now move the binary files into their own area, and delete them from the text file area:
$ cd ~atp/e/
$ sudo tar cf ~atp/e/etc-imp-bin.tar -C ~atp/e/etc FILES
$ mkdir ~atp/e/etc-kb
$ cd ~atp/e/etc-kb
$ sudo tar -xpf ~atp/e/etc-imp-bin.tar

$ cd ~atp/e/etc
$ sudo rm -rf FILES
Finally, import both sets of files with the appropriate cvs -k options:
$ cd ~atp/e/etc
$ sudo cvs import -ko -m "Importing everything in /etc as of 2002-12-05, almost all original unchanged files, principally from Debian 3.0, also Oracle and a few of my own scripts." unix-koudelka/etc Debian slash-etc-original

$ cd ~atp/e/etc-kb
sudo rm .pwd.lock  ;# Delete files we don't want under CVS.
sudo cvs import -kb -m "Importing everything in /etc as of 2002-12-05, almost all original unchanged files, principally from Debian 3.0, also Oracle and a few of my own scripts." unix-koudelka/etc Debian slash-etc-original
Check permissions on sensitive files in the CVS repository:

This only applies if you did the cvs import as root rather than a non-priviliged user. Do e.g.:
sudo chmod 400 /home/cvsroot/unix-koudelka/etc/*hadow*

TODO: Write a script to look at the permissions in /etc/ and set the permissions in the CVS respository the same way. --atp@piskorski.com, 2002/12/05 16:39 EST

Optionally, double check that all files were imported:

How might we programatically verify that we got everything, that the import didn't stop three quarters of the way through or something? Well, here's one half-hearted check - we'll look for all the directories beneath /etc, and all the directories that we just checked into CVS:
$ find /etc -type d -print | sort > ~/t/1.txt $ find /home/cvsroot/unix/etc -name CVS -prune -o -type d -print | sed 's/\/home\/cvsroot\/unix//' | sort > ~/t/2.txt

The two files 1.txt and 2.txt should be identical.

Prepare to put the CVS directories into /etc/:

Now we have all our normal files in the repository, but there may be lots of other stuff in /etc (symlinks, FIFOs, binaries, etc.) which we have not put into CVS, and we don't want to screw any of that stuff up. For example, we do not want to change the permissions of any of the files already in /etc. Therefore, what we're going to do, is create a mirror tree of all the directories in /etc, but with nothing in them except the contents of the CVS directories. Then we can copy each of the CVS directories into the real /etc, and /etc will auto-magically becoming a CVS working copy.
So, let's checkout a working copy of the /etc stuff, then delete all the normal files, except for any files in the CVS directories:
$ mkdir /tmp/atp ; cd /tmp/atp $ cvs co -d etc1 unix/etc $ sudo find etc1 -name CVS -prune -o -type f -print | sudo xargs rm

xargs does not properly handle filenames containing whitespace, so if you have such filenames (e.g., "/etc/kde2/colors/40 Colors"), instead of the above command use:
$ sudo find etc1 -name CVS -prune -o -type f -exec rm {} \;

Verify that /tmp/atp/etc1 really is just the CVS directories. E.g., this should not print out anything at all:
$ find etc1 ! -type d | sed '/\/CVS\/Root$/d' | sed '/\/CVS\/Repository$/d' | sed '/\/CVS\/Entries$/d'

(Below, it will turn out that the above rm of the non-CVS files isn't really necessary. But it doesn't hurt.)
Now, let's change the owner of all the CVS directories to how we'll want it in the real /etc:
$ sudo find etc1 $ -type d -a -name CVS $ -exec chown -R root:staff {} \;
$ sudo find etc1 $ -type d -a -name CVS $ -exec chmod 775 {} \;
$ sudo find etc1 $ -name Entries -o -name Repository -o -name Root $ -print | sudo xargs chmod 664
Note that we didn't change the owner of the non-CVS directories, because we must not copy them into /etc. And leaving the owner of e.g. /tmp/atp/etc1/inet/ set to "andy" will help make it more obvious if we screw up and copy that inet directory into /etc/.
Before we do anything else, let's make a backup of the real /etc, just in case. Note that you want to use GNU tar for this:
$ sudo gtar cf /tmp/atp/etc-backup.tar -C /etc /etc

Now that you have tarball backup of /etc, un-tar a copy, pretend it's the real /etc, and practice what we're going to do next.
$ cd /tmp/atp/ $ sudo gtar xpf etc-backup.tar

Create this script and name it "cp-cvs-etc.tcl":
#!/bin/sh
# Restart using tclsh: \
exec tclsh "$0" "$@"
#
# cp-cvs-etc.tcl - Copy only CVS directories - used to help put /etc
# under CVS control.

set path_tmp [lindex $argv 0]  ;# E.g.: /tmp/atp/etc1
set path_etc [lindex $argv 1]  ;# /etc

# Normalize incoming paths to NOT have trailing slash, otherwise the
# regexp below will fail:
foreach var [list path_tmp path_etc] {
   regsub -- {/+$} [set $var] {} $var
}

set io_id [open "|find $path_tmp -name CVS -print" r]
while { [gets $io_id line] >= 0 } {
   if { [regexp "$path_tmp/(.*)" $line match relpath] } {
      exec cp -pr $line $path_etc/$relpath
   } else {
      puts "ERROR:  Regexp failed for '$line'."
   }
}
if { [catch {close $io_id} std_error] } {
   puts "Standard Error output:\n${std_error}"
}
And then test it with the un-tarred copy of the real /etc you put into /tmp/atp/etc :
$ sudo ./cp-cvs-etc.tcl /tmp/atp/etc1 /tmp/atp/etc

And then do it for real:
$ sudo ./cp-cvs-etc.tcl /tmp/atp/etc1 /etc

We're done! Go into /etc and take a look.

Putting new stuff in `/etc` under CVS

Using the non-invasive method:

If you personally write new scripts and put them into /etc, simply do a cvs add and cvs commit. If you install new software which creates new files in /etc, you can either just cvs add them the same way (by far the simplest), or you can cvs import it instead (slicker but requiring much more effort).
Assuming you want to do it the cvs import way, here's an example, taken from when I installed he PostgreSQL packages on a Debian 3.0 box:
Below, FILES is the list of new files in /etc/, and FILE_NO_CVS_DIRS is the list of new directories in /etc/ which do not yet have their proper CVS subdirectories, in the case of this example:
FILES:
  postgresql/
  cron.d/postgresql
  init.d/postgresql
  logcheck/ignore.d.paranoid
  logcheck/violations.ignore.d
FILE_NO_CVS_DIRS:
  postgresql/
  logcheck/ignore.d.paranoid
  logcheck/violations.ignore.d
Here are all the steps, quite similar to when we imported everything in /etc into CVS for the first time, above:
$ mkdir ~atp/e ; cd ~atp/e/ 
$ sudo tar cf ./etc-backup.tar -C /etc /etc 
$ sudo tar xpf etc-backup.tar 
$ sudo tar cf ./etc-imp-new.tar -C ./etc  FILES
$ mkdir etc-new ; cd etc-new
$ sudo tar -xpf ../etc-imp-new.tar 

$ sudo cvs import -ko  -m "Importing Debian PostgreSQL packages /etc stuff, 2002-12-06."  unix-koudelka/etc Debian slash-etc-original

$ mkdir /tmp/atp ; cd /tmp/atp 
$ cvs co -d etc1 unix-koudelka/etc 
$ sudo find etc1 -name CVS -prune -o -type f -exec rm {} \;
$ sudo tar cf ./etc-cvs-dirs-new.tar -C ./etc1  FILE_NO_CVS_DIRS
$ mkdir etc-cvs-dirs-new ; cd etc-cvs-dirs-new
$ sudo tar -xpf ../etc-cvs-dirs-new.tar

$ sudo find . $ -type d -a -name CVS $ -exec chown -R root:staff {} \;
$ sudo find . $ -type d -a -name CVS $ -exec chmod 775 {} \;
$ sudo find . $ -name Entries -o -name Repository -o -name Root $ -print | sudo xargs chmod 664

$ sudo ~atp/bin/cp-cvs-etc.tcl /tmp/atp/etc-cvs-dirs-new /etc 
$ cd /etc
$ sudo -q update -d -p
TODO: There's one problem with the above method. The cp-cvs-etc.tcl script properly creates the new CVS subdirectories, but cvs update will not properly update the CSV/Entries file of the parent directories. How to work around this?

atp@piskorski.com

$Id: cvs-unix.html,v 1.4 2003/01/31 23:50:30 andy Exp $

Version Control - UNIX configuration files

Companion Documents

Introduction

Putting Oracle 8.1.7 under CVS

Other Oracle files you should probably also put under CVS:

Using the traditional CVS import/checkout method:

Alternately, using the non-invasive method:

Putting /etc under CVS

Using the non-invasive method:

Do cvs import as root or not?

Prepare to do the cvs import:

Do the import, using -kb for everything:

Or, instead do the import in two steps, using -ko and -kb:

Check permissions on sensitive files in the CVS repository:

Optionally, double check that all files were imported:

Prepare to put the CVS directories into /etc/:

Putting new stuff in /etc under CVS

Using the non-invasive method:

Putting `/etc` under CVS

Putting new stuff in `/etc` under CVS