Using Subversion to maintain programs

I am using subversion to keep all my shell scripts, it helps to maintain the scripts change history:

To use subversion, first we need to create a repository, on svn server

# svnadmin create /usr/local/scripts
cat <<EOF > /usr/local/scripts/conf/svnserve.conf
anon-access = none
auth-access = write
password-db = passwd
EOF

The rest of the parts are all done on clients:
To add files to subversion repository, we can run

svn import /usr/local/scripts/check_mem.sh \
svn+ssh://lnxscrpt@linux/usr/local/scripts/check_mem.sh
Adding         /usr/local/scripts/check_mem.sh

Alternatively, we can check out an SVN copy, update on our local copy, then check in whatever changes done on local copy to SVN repository
1. To check out

$ mkdir ~/work; cd ~/work
$ svn co svn+ssh://lnxscrpt@linux/usr/local/scripts
Checked out revision 0.

We will have ~/work/scripts directory now

After we check out, other people may have made some changes to the repository, it's always a good practice to update our local copy before changing anything.

$ svn up scripts
At revision 0.

2. To add check_load.sh, copy check_load.sh to ~/work/scripts, and run

$  svn add scripts/check_load.sh
A         scripts/check_load.sh

3. Commit our changes:

$ svn ci scripts/
Adding         scripts/check_load.sh
Transmitting file data .
Committed revision 1.

4. Make sure check_load.sh is in SVN repository:

$  svn ls svn+ssh://lnxscrpt@linux/usr/local/scripts
check_load.sh
check_mem.sh

5. check svn log

$ svn log svn+ssh://lnxscrpt@linux/usr/local/scripts/check_load.sh
------------------------------------------------------------------------
r1 | lnxscrpt | 2013-09-03 14:25:11 +0800 (Tue, 03 Sep 2013) | 2 lines

added check_load.sh at 2:25pm

------------------------------------------------------------------------

$ svn log svn+ssh://lnxscrpt@linux/usr/local/scripts/check_mem.sh
------------------------------------------------------------------------
r5 | lnxscrpt | 2013-09-03 14:44:38 +0800 (Tue, 03 Sep 2013) | 2 lines

import check_mem.sh

------------------------------------------------------------------------

Note: whenever we svn commands, we will be prompted the password for lnxscrpt@linux, we can set up password ssh login by following: http://linuxscripter.blogspot.com/2012/05/set-up-password-less-ssh-login-using.html

Set up password-less SSH login using ssh-keygen and ssh-copy-id

Logging in remote systems without password is very useful in system administration. It can be used for executing command on remote systems, copying files to other systems using rsync/rdist.

For the past few years, I have been manually setting up the key trust between systems. Here are the steps I followed.

Suppose I need to login to localserver from remoteserver using account scripter without password.

1. generate public-private key pair on localserver

$ ssh-keygen -t dsa

    you can use either dsa or rsa.

2. copy scripter's public key to remoteserver

$ scp id_dsa.pub remoteserver:

3. on remoteserver, append the public key copied in step 2 to ~scripter/.ssh/authorized_keys

$ cat id_dsa.pub >> ~scripter/.ssh/authorized_keys

    if ~scripter/.ssh doesn't exist, manually create it

4. on localserver the directory and files are auto-generated, so the permissions are correct, on remoteserver, as we manually created the directories, we need to make the permissions are correct.

$ chmod 700 ~scripter/.ssh

$ chmod 600 ~scripter/.ssh/authorized_keys

After all these steps, we can login from localserver to remoteserver without entering password:

$ ssh remoteserver hostname

remoteserver

The steps are quite easy, but can it be easier?
Yes! recently I learned the script ssh-copy-id from a forum post, to set up key trust from localserver to remoteserver using ssh-keygen and ssh-copy-id
1. generate public-private key pair on localserver

$ ssh-keygen -t dsa

2. copy the public key from localserver to remoteserver, and set up the key trust

$ ssh-copy-id -i ~scripter/.ssh/id_dsa.pub \
scripter@remoteserver

After you entering the password, everything is set up for you, you can log in remotely without password

$ ssh remoteserver hostname

remoteserver

ssh-copy-id saves us all the troubles of creating files and setting up the proper permission :)

Copy files from one server to another

In my work, I often need to copy files or whole directories between different servers. These are the ways I used often:

1. scp

scp -r directory remote:/path/to/directory

2. tar + ssh

tar -zcpf - directory | ssh remote tar -zxpf \
-C /path/to/directory

in Solaris, there is no -C -z flag, so I use:

tar -cpf - directory | ssh remote \
"(cd /path/to/directory; tar -xpf -)"

3. rsync over ssh
I have a central server, everyday there are some new files generated in one directory. I need to push this directory to remote servers everyday.
It's possible to schedule some cronjob to scp or tar + ssh this directory, But it will copy the whole directory everyday. To use rsync, we only need to copy the changes to this directory.

rync -e ssh -az --delete directory \
remote:/path/to/directory

4. nc + tar
A few years ago, I need to copy huge amount of data to a remote data center in another country, while exploring ways to copy the data, I found this utility nc.
But nc doesn't encrypt the data, I only tested it within the same data center.

on remote server:

cd /path/to/directory

nc -l -p 8888 | tar -zxvf -

on source server:

tar -czf - directory | nc remote 8888

on the remote server, I used -v flag, so I could monitor the copying progress.