Tuesday, 10 September 2013

How to install Zabbix server on Linux server

Zabbix is an open source monitoring server, besides monitoring the service status, it also stores the historical status of service, so we can generate graphs easily from Zabbix.  I have been using Nagios for many years, recently I decided to give Zabbix a try.

Here are the steps I installed server.

First we need to download zabbix software, current stable version is 2.0.8, so I downloaded zabbix-2.0.8.tar.gz

1. create OS account for running zabbix server
# useradd zabbix
2.
# tar -zxpf zabbix-2.0.8.tar.gz
# cd zabbix-2.0.8

3. Zabbix uses databases as its data storage, it supports MySQL, PostgreSQL, Oracle, DB2 and SQLite. In my setup I use MySQL.
Before Installing zabbix, we need to configure the database properly.
Follow instructions on this link to setup the database https://www.zabbix.com/documentation/2.0/manual/appendix/install/db_scripts
For MySQL:
mysql> create database zabbix character set utf8 collate utf8_bin;
mysql> grant all privilegs on zabbix.* to 'zabbix'@'localhost' identified by 'zabbix';
mysql> exit;
# mysql -uzabbix -pzabbix zabbix < database/mysql/schema.sql
# mysql -uzabbix -pzabbix zabbix < database/mysql/images.sql
# mysql -uzabbix -pzabbix zabbix < database/mysql/data.sql


4.
# ./configure --enable-server --with-mysql --with-net-snmp \
  --with-libcurl --prefix=/usr/local/zabbix

# make install


5. start zabbix server
# /usr/local/zabbix/sbin/zabbix_server

6. Install zabbix web interface:
# mkdir /var/www/html/zabbix/
# cp -pr frontends/php/* /var/www/html/zabbix/

7. Installing zabbix frontend
Open browser, go to http://localhost/zabbix, follow the instructions for frontend installation wizard to complete the installation.

8. After Installing, you will be able to login zabbix system, the default login ID is Admin, password is zabbix.

Next we need to install zabbix agent on remote hosts and zabbix server to monitor remote hosts.

Reference: https://www.zabbix.com/documentation/2.0/manual/installation/install#from_the_sources

Tuesday, 3 September 2013

Using Subversion to maintain programs

I am using subversion to keep all my shell scripts, it helps to maintain the scripts change history:

To use subversion, first we need to create a repository, on svn server
# svnadmin create /usr/local/scripts
cat <<EOF > /usr/local/scripts/conf/svnserve.conf
anon-access = none
auth-access = write
password-db = passwd
EOF

The rest of the parts are all done on clients:
To add files to subversion repository, we can run
svn import /usr/local/scripts/check_mem.sh \
svn+ssh://lnxscrpt@linux/usr/local/scripts/check_mem.sh

Adding         /usr/local/scripts/check_mem.sh
Alternatively, we can check out an SVN copy, update on our local copy, then check in whatever changes done on local copy to SVN repository
1. To check out
$ mkdir ~/work; cd ~/work
$ svn co svn+ssh://lnxscrpt@linux/usr/local/scripts
Checked out revision 0.

We will have ~/work/scripts directory now

After we check out, other people may have made some changes to the repository, it's always a good practice to update our local copy before changing anything.
$ svn up scripts
At revision 0.

2. To add check_load.sh, copy check_load.sh to ~/work/scripts, and run
$  svn add scripts/check_load.sh
A         scripts/check_load.sh

3. Commit our changes:
$ svn ci scripts/
Adding         scripts/check_load.sh
Transmitting file data .
Committed revision 1.

4. Make sure check_load.sh is in SVN repository:
$  svn ls svn+ssh://lnxscrpt@linux/usr/local/scripts
check_load.sh
check_mem.sh

5. check svn log
$ svn log svn+ssh://lnxscrpt@linux/usr/local/scripts/check_load.sh
------------------------------------------------------------------------
r1 | lnxscrpt | 2013-09-03 14:25:11 +0800 (Tue, 03 Sep 2013) | 2 lines

added check_load.sh at 2:25pm

------------------------------------------------------------------------

$ svn log svn+ssh://lnxscrpt@linux/usr/local/scripts/check_mem.sh
------------------------------------------------------------------------
r5 | lnxscrpt | 2013-09-03 14:44:38 +0800 (Tue, 03 Sep 2013) | 2 lines

import check_mem.sh

------------------------------------------------------------------------

Note: whenever we svn commands, we will be prompted the password for lnxscrpt@linux, we can set up password ssh login by following: http://linuxscripter.blogspot.com/2012/05/set-up-password-less-ssh-login-using.html

Monday, 2 September 2013

Using Nagios and NRPE to monitor remote hosts

We have setup Nagios to monitor our localhost centos, but how to monitor remote hosts? We can monitor remote hosts using SNMP or NRPE
In our testing environment, we will setup Nagios to monitor host centos-1 and centos-2 using NRPE.
To use NRPE, we need to install NRPE on both monitoring host and remote hosts:

Steps to setup remote hosts for NRPE monitoring:
1. Download nagios-plugins-1.4.16.tar.gz, nrpe-2.14.tar.gz
2. Create account for NRPE service
# useradd nagios
# passwd nagios

3. Install nagios-plugs
# tar zxpf nagios-plugins-1.4.16.tar.gz
# cd nagios-plugins-1.4.16
# ./configure --with-nagios-user=nagios \
  --with-nagios-group=nagios

# make
# make install
# chown nagios:nagios /usr/local/nagios

4. Install NRPE
# tar zxpf nrpe-2.14.tar.gz
# cd nrpe-2.14
# ./configure
# make all
# make install-plugin
# make install-daemon
# make install-daemon-config

5. Modify /usr/local/nagios/etc/nrpe.cfg to allow host centos(192.168.100.100) to connect to NRPE
allowed_hosts=127.0.0.1,192.168.100.100

6. Start NRPE, enable autostart to NRPE
# /usr/local/nagios/bin/nrpe -c \
  /usr/local/nagios/etc/nrpe.cfg -d

# echo '/usr/local/nagios/bin/nrpe -c \
  /usr/local/nagios/etc/nrpe.cfg -d' >> /etc/rc.local

7. Verify NRPE is working properly
# /usr/local/nagios/libexec/check_nrpe -H localhost
NRPE v2.14

Steps to setup monitoring host for NRPE monitoring
1. Download nrpe-2.14.tar.gz
2. Install NRPE
# tar -zxpf nrpe-2.14.tar.gz
# cd nrpe-2.14
# ./configure
# make all
# make install-plugin

3. Enable monitoring of centos-1
Update /usr/local/nagios/etc/objects/commands.cfg to enable command check_nrpe
# ‘check_nrpe’ command definition
define command{
        command_name check_nrpe
        command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -t 30 -c $ARG1$
}

update /usr/local/nagios/etc/nagios.cfg to include this line
cfg_file=/usr/local/nagios/etc/objects/centos-1.cfg

4. Edit /usr/local/nagios/etc/objects/centos-1.cfg
define host{
        use                 linux-server
        host_name           centos-1
        alias               centos-1
        address             192.168.100.101
        }


define hostgroup{
        hostgroup_name      centos
        alias               centos
        members             centos-1
        }


define service{
        use                 generic-service
        host_name           centos-1
        service_description PING
        check_command       check_ping!100.0,20%!500.0,60%
        }

define service{
        use                 generic-service
        host_name           centos-1
        service_description Server Load
        check_command       check_nrpe!check_load
        }

define service{
        use                 generic-service
        host_name           centos-1
        service_description /rhiso Usage
        check_command       check_nrpe!check_rhiso
        }

5. Restart Nagios service
# service nagios restart

Steps to enable nrpe commands check_load and check_rhiso on remote host centos-1
Edit /usr/local/nagios/etc/nrpe.cfg, add these 2 lines:
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
command[check_rhiso]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% /rhiso

restart nrpe daemon
# pkill nrpe
# /usr/local/nagios/bin/nrpe -c \
  /usr/local/nagios/etc/nrpe.cfg -d

Login to http://centos/nagios again, you will see now we have host centos-1 monitored, similarly we can setup the monitoring of centos-2

Setting up Nagios monitoring server

Nagios is an open-source monitoring software, you can monitor different host resources like host UP/down status, disk usage, system load, logfile.
Setting up Nagios on Linux is very straight forward.

In our test environment, we will setup host centos as our monitoring host.

Steps to install Nagios on centos:
1. Download nagios core, nagios plugins
http://www.nagios.org/download/core, choose DIY option.
http://www.nagios.org/download/plugins, download latest stable version.

2. Create nagios account:
# useradd nagios
# passwd nagios
# groupadd nagcmd
# usermod -a -G nagcmd nagios
# usermod -a -G nagcmd apache

3. Install Nagios
# tar -zxpf nagios-3.5.0.tar.gz
# cd nagios
# ./configure --with-command-group=nagcmd
# make all
# make install
# make install-init
# make install-config
# make install-commandmode
# make install-webconf

4. Create nagiosadmin account, this account will be used to login to nagios web interface
# htpasswd -c /usr/local/nagios/etc/htpasswd.users \
  nagiosadmin


5. Edit /etc/httpd/conf.d/nagios.conf, pointing AuthUserFile to /usr/local/nagios/etc/htpasswd.users

6. Start apache
# service httpd restart

7. Install Nagios Plugins
# tar -zxpf nagios-plugins-1.4.16.tar.gz
# cd nagios-plugins-1.4.16
# ./configure --with-nagios-user=nagios \
  --with-nagios-group=nagios

# make; make install

8. Enable autostart of Nagios and start Nagios service
# chkconfig --add nagios
# chkconfig nagios on

# service nagios start
9. Login to http://centos/nagios, you will see we already have centos monitored by Nagios


In the next post http://linuxscripter.blogspot.com/2013/09/using-nagios-and-nrpe-to-monitor-remote.html, I will discuss how to monitor remote hosts using Nagios and NRPE.

Reference: http://nagios.sourceforge.net/docs/3_0/quickstart-fedora.html

Wednesday, 26 June 2013

How to setup MySQL replication

MySQL replication is the technique to replicate one mysql database (master) to one or more databases (slave), using replication, we can:

1) Separate read and write
Write is only done on master server, all the read operations are done one the slave servers. We can also run expensive report on slave servers without affecting the performance of master and other slaves.

2) backup.
MySQL stores all the files used for database/tables in data/ directory, to backup mysql, we can backup the data/ directory. However backing up data/ while MySQL is running will give us an inconsistent copy of data/. To make consistent backup, we need to stop MySQL first, and this will cause downtime.
In MySQL replication environment, we can stop one slave and do the backup on slaves. Master and other slaves are still online during our backup.


To setup MySQL replication, each server in the replication needs to have a unique server-id. The valid range of server-id is 0 to 2^32-1, but replication only allows positive server-id.

I have 3 servers called: centos, centos-1, centos-2. All servers have MySQL freshly installed.

[root@centos ~]# getent hosts | grep centos
192.168.100.100 centos
192.168.100.101 centos-1
192.168.100.102 centos-2

Below are the steps I did to setup replication from centos to centos-1 and centos-2.
1. update /etc/my.cnf and restart MySQL
on centos:
[root@centos ~]# cat <<EOF > /etc/my.cnf
[mysqld]
server-id = 100
log-bin = master-bin.log 
EOF 
[root@centos ~]# service mysql restart

on centos-1:
[root@centos-1 ~]# cat <<EOF > /etc/my.cnf
[mysqld]
server-id = 101
relay-log-index = slave-relay-bin.index
relay-log = slave-relay-bin
replicate-wild-ignore-table=mysql.%
replicate-wild-ignore-table=information_schema.%
replicate-wild-ignore-table=performance_schema.%
EOF
[root@centos-1 ~]# service mysql restart

on centos-2:
[root@centos-2 ~]# cat <<EOF > /etc/my.cnf
[mysqld]
server-id = 102
relay-log-index = slave-relay-bin.index
relay-log = slave-relay-bin
replicate-wild-ignore-table=mysql.%
replicate-wild-ignore-table=information_schema.%
replicate-wild-ignore-table=performance_schema.%
EOF
[root@centos-2 ~]# service mysql restart

2. Get master bin log and position.
mysql> flush table with read lock;
Query OK, 0 rows affected (0.00 sec)

mysql> show master status;

+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| master-bin.000007 |      272 |              |                  |
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)


3. On master, create account used for replication. Slave uses this account to connect to master. Multiple clients can share the same account, we can also create one account for each slave. This account need to have privilege "REPLICATION SLAVE".
mysql> CREATE USER 'repl'@'%' IDENTIFIED BY 'slavepass';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';
mysql> FLUSH PRIVILEGES;

4. Setup slave using the information found in step 2 and 3.
On both centos-1 and centos-2
mysql> change master to
    -> master_host='192.168.100.100',
    -> master_port=3306,
    -> master_user='rep1',
    -> master_password='slavepass',
    -> master_log_file='master-bin.000007',
    -> master_log_pos=272;
mysql> start slave;
5. Confirm replication is working:
[root@centos-1 data]# tail -1 centos-1.err
130626  9:47:16 [Note] Slave I/O thread: connected to master 'rep1@192.168.100.100:3306',replication started in log 'master-bin.000007' at position 272
[root@centos-2 data]# tail -1 centos-2.err
130626  9:48:04 [Note] Slave I/O thread: connected to master 'rep1@192.168.100.100:3306',replication started in log 'master-bin.000007' at position 272

we can also insert some data on master, querying on any slave will show us the data we inserted on master.

We are done! MySQL replication is up and running.

reference:
http://dev.mysql.com/doc/refman/5.6/en/replication-howto.html
http://top-performance.blogspot.com/2012/03/how-to-setup-mysql-replication-in-11.html

Sunday, 26 May 2013

bash string manipulation

Bash supports many ways of string manipulation, it includes substring, substring replacement, substring removal, string length.

In how to rename multiple files in one command, one of the ways I shared to rename *.txt to *.sh is

for f in *.txt
do
        mv $f ${f/.txt/.sh}
done

How ${f/.txt/.sh} is doing the magic? Actually this is string replacement in bash.

${f/.txt/.sh} means replacing ".txt" in string $f with ".sh", so for f=abc.txt, ${f/.txt/.sh} will produce abc.sh

[linuxscripter@localhost ~]$ f=abc.txt
[linuxscripter@localhost ~]$ echo $f{f/.txt/.sh}
abc.sh

But the example I gave in rename multiple files in one command was not so correct.
For f=abc.txt.txt, ${f/.txt/.sh} will produce abc.sh.txt instead of abc.txt.sh

[linuxscripter@localhost ~]$ f=abc.txt.txt
[linuxscripter@localhost ~]$ echo $f{f/.txt/.sh}
abc.txt.sh

To correct this, we need to use ${f/%.txt/.sh}

[linuxscripter@localhost ~]$ f=abc.txt.txt
[linuxscripter@localhost ~]$ echo $f{f/%.txt/.sh}
abc.txt.sh

The syntax of string replacement includes:
1. ${var/Pattern/Replacement}   : replacing the first match of "Pattern" with "Replacement"
2. ${var//Pattern/Replacement}  : replacing every match of "Pattern" with "Replacement"
3. ${var/#Pattern/Replacement}  : if $var starts with "Pattern", replace "Pattern" with "Replacement"
4. ${var/%Pattern/Replacement}  : if $var ends with "Pattern", replace "Pattern" with "Replacement"

Some examples will make the syntax clearer.

[linuxscripter@localhost ~]$ f=abc.txt.abc.txt

[linuxscripter@localhost ~]$ ### testing syntax 1
[linuxscripter@localhost ~]$ echo ${f/txt/sh}
abc.sh.abc.txt

[linuxscripter@localhost ~]$ ### testing syntax 2
[linuxscripter@localhost ~]$ echo ${f//txt/sh}
abc.sh.abc.sh

[linuxscripter@localhost ~]$ ### testing syntax 3, [linuxscripter@localhost ~]$ echo ${f/#txt/sh}
abc.txt.abc.txt

[linuxscripter@localhost ~]$ ### testing syntax 3,

[linuxscripter@localhost ~]$ echo ${f/#abc/sh}
sh.txt.abc.txt

[linuxscripter@localhost ~]$ ### testing syntax 4,
[linuxscripter@localhost ~]$ echo ${f/%abc/sh}
abc.txt.abc.txt
[linuxscripter@localhost ~]$ ### testing syntax 4
[linuxscripter@localhost ~]$ echo ${f/%txt/sh}
abc.txt.abc.sh

At first these syntax may not appear very intuitive, I found this method help to understand and memorize them.
Variable is referred by add '$' infront of it
1. '/' is used for replacement, same as what we do in awk and sed
2. Once we know '/' is for replacement, '//' should be too difficult for us.
3. On keyboard, '#' is at the left of '$', so '#' is used to replace the left side of the variable.
4. On keyboard, '%' is at the right of '$', so '%' is used to replace the right side of the variable.

Besides string replacement, of course there are a lot more bash can do.
${#var} returns length of $var
[linuxscripter@localhost ~]$ echo {#f}
15

${var:pos:len} returns substring of $var, starting at position "pos", with length upto "len", if len is empty or len is too big, return substring starting at position "pos" until to the end of the string.
[linuxscripter@localhost ~]$ echo ${f:2:5} ${f:2:100} ${f:2}
c.txt c.txt.abc.txt c.txt.abc.txt

${var#substring} delete shortest match of "substring" from front of $var
[linuxscripter@localhost ~]$ echo ${f#abc}
.txt.abc.txt
[linuxscripter@localhost ~]$ echo ${f#*txt}
.abc.txt

${var##substring} delete longest match of "substring" from front of $var
[linuxscripter@localhost ~]$ echo ${f##*abc}
.txt

${var%substring} delete longest match of "substring" from back of $var
[linuxscripter@localhost ~]$ echo ${f%txt}
abc.txt.abc.
[linuxscripter@localhost ~]$ echo ${f%abc*}
abc.txt.

${var%%substring} delete longest match of "substring" from back of $var
[linuxscripter@localhost ~]$ echo ${f%%txt*}
abc.

http://tldp.org/LDP/abs/html/string-manipulation.html
http://bbs.chinaunix.net/forum.php?mod=viewthread&tid=218853&page=7

Friday, 17 May 2013

Use HTML pre tag to keep email display format

In my servers, I have some shell scripts to send system status reports to my email address.
It's very easy to implement, just run script to generage the report and send the report using sendmail, mail, or mailx.

e.g. if I need to email the output of w, I can simply put it in this way:
w | mail linuxscripter@myemail.com

but it doesn't have any subject, To fields.

Personally I prefer sendmail, so I added the email headers in this way.
(cat <<EOF
From: linuxscripter@myserver.com
To: linuxscripter@myemail.com
Subject: output of w

EOF
w)| sendmail -t

This looked pretty ok, and the email displayed nicely in my outlook express 5.

But in 2006, my company upgraded email client to outlook express 6, all the report format were gone, things should appear in the same column didn't.

How should I preserve the report format when displayed in outlook express 6?
I worked as a PHP web developer for 3 years, so the first thing came to my mind is html <pre> tag.
To use html tag in email, I need to set the email MIME type as text/html

This was how I did.
(cat <<EOF
From: linuxscripter@myserver.com
To: linuxscripter@myemail.com
Subject: output of w
Content-Type: text/html;

EOF
echo "<html><pre>"; w; echo "</pre></html>")| sendmail -t

My email reports became nicely formatted again!