Tuesday 24 September 2013

Using tmpfs to improve PostgreSQL performance

In PostgreSQL, the statistics collector collects the following information:
  1. statistics about table and index accesses
  2. usage of user-defined functions
  3. current command executed by any server process.

The statistics collector then passes the information to backends using temporary files, location of the temporary files are defined by stats_temp_directory in postgresql.conf, by defaults it points to $PGDATA/pg_stat_tmp
As PostgreSQL is running, there are continuous I/O in stats_temp_directory, the disk IO may affect database performance. PostgreSQL recommends to point stats_temp_directory to RAM-based file system, decreasing physical I/O, thus increasing database performance.

We can use either ramfs or tmpfs, for the differences of the two, see http://www.thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux/

I will use tmpfs for stats_temp_directory, here are my steps
1. create directory for our new mount point:
# mkdir /mnt/tmp

2. mount tmpfs
# mount -t tmpfs tmpfs /mnt/tmp/ -o size=200m
now we have a file system of 200M.
# df -h /mnt/tmp/
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 200M  8.0K  200M   1% /mnt/tmp
every time the server reboots, /mnt/tmp will be gone, to make the configuration persistent, add this line to /etc/fstab
tmpfs    /mnt/tmp    tmpfs   defaults,size=200m    0 0


3. edit postgres.conf, add this line:
stats_temp_directory = '/mnt/tmp'

4. restart database
$ pg_ctl -D /var/lib/pgsql/data restart

5. confirm we are using the tmpfs
[postgres@linux ~]$ psql
psql (8.4.11)
Type "help" for help.

postgres=# show stats_temp_directory;
 stats_temp_directory
----------------------
 /mnt/tmp
(1 row)

postgres=# \q
[postgres@linux ~]$ ls -lh /mnt/tmp/
total 8.0K
-rw------- 1 postgres postgres 6.0K Sep 24 13:41 pgstat.stat
[postgres@linux ~]$
Similarly, we can use tmpfs to improve the performance of Nagios server.
reference:
http://www.postgresql.org/docs/9.1/static/monitoring-stats.html

Tuesday 10 September 2013

How to install Zabbix agent and monitor remote linux servers

I installed Zabbix server in my previous post: http://linuxscripter.blogspot.com/2013/09/how-to-install-zabbix-server-on-linux.html
To monitor remote servers, we can install Zabbix agent.
Installing Zabbix agent is quite straight forward

Here are my installation steps:
1. create user account for zabbix
# useradd zabbix
2. download current stable version zabbix-2.0.8.tar.gz from www.zabbix.com
3. configure and install zabbix agent
# tar -zxpf zabbix-2.0.8.tar.gz
# cd zabbix-2.0.8
#  ./configure --enable-agent --prefix=/usr/local/zabbix
# make install
4. modify /usr/local/zabbix/etc/zabbix_agentd.conf
In previouse post, we installed zabbix server on 192.168.100.100, so modify the Server parameter
Server=192.168.100.100
5. restart zabbix_agentd to reload the new configuration file
now we can monitor this host from zabbix server.

To monitor remote host, all the configuration are done in the Zabbix web interface.
1. open web browser, go to http://zabbix-server/zabbix
2. Configuration -> Hosts -> Create host
Enter the host name, IP address, choose "Linux servers" as the host group.


3. After saving, you will see the host linux-1 is monitored by zabbix now.


 4. At the monitoring dashboard, you will see issues are shown in different colors


Reference: https://www.zabbix.com/documentation/2.0/manual/installation/install#from_the_sources

How to install Zabbix server on Linux server

Zabbix is an open source monitoring server, besides monitoring the service status, it also stores the historical status of service, so we can generate graphs easily from Zabbix.  I have been using Nagios for many years, recently I decided to give Zabbix a try.

Here are the steps I installed server.

First we need to download zabbix software, current stable version is 2.0.8, so I downloaded zabbix-2.0.8.tar.gz

1. create OS account for running zabbix server
# useradd zabbix
2.
# tar -zxpf zabbix-2.0.8.tar.gz
# cd zabbix-2.0.8

3. Zabbix uses databases as its data storage, it supports MySQL, PostgreSQL, Oracle, DB2 and SQLite. In my setup I use MySQL.
Before Installing zabbix, we need to configure the database properly.
Follow instructions on this link to setup the database https://www.zabbix.com/documentation/2.0/manual/appendix/install/db_scripts
For MySQL:
mysql> create database zabbix character set utf8 collate utf8_bin;
mysql> grant all privilegs on zabbix.* to 'zabbix'@'localhost' identified by 'zabbix';
mysql> exit;
# mysql -uzabbix -pzabbix zabbix < database/mysql/schema.sql
# mysql -uzabbix -pzabbix zabbix < database/mysql/images.sql
# mysql -uzabbix -pzabbix zabbix < database/mysql/data.sql


4.
# ./configure --enable-server --with-mysql --with-net-snmp \
  --with-libcurl --prefix=/usr/local/zabbix

# make install


5. start zabbix server
# /usr/local/zabbix/sbin/zabbix_server

6. Install zabbix web interface:
# mkdir /var/www/html/zabbix/
# cp -pr frontends/php/* /var/www/html/zabbix/

7. Installing zabbix frontend
Open browser, go to http://localhost/zabbix, follow the instructions for frontend installation wizard to complete the installation.

8. After Installing, you will be able to login zabbix system, the default login ID is Admin, password is zabbix.

Next we need to install zabbix agent on remote hosts and zabbix server to monitor remote hosts.

Reference: https://www.zabbix.com/documentation/2.0/manual/installation/install#from_the_sources

Tuesday 3 September 2013

Using Subversion to maintain programs

I am using subversion to keep all my shell scripts, it helps to maintain the scripts change history:

To use subversion, first we need to create a repository, on svn server
# svnadmin create /usr/local/scripts
cat <<EOF > /usr/local/scripts/conf/svnserve.conf
anon-access = none
auth-access = write
password-db = passwd
EOF

The rest of the parts are all done on clients:
To add files to subversion repository, we can run
svn import /usr/local/scripts/check_mem.sh \
svn+ssh://lnxscrpt@linux/usr/local/scripts/check_mem.sh

Adding         /usr/local/scripts/check_mem.sh
Alternatively, we can check out an SVN copy, update on our local copy, then check in whatever changes done on local copy to SVN repository
1. To check out
$ mkdir ~/work; cd ~/work
$ svn co svn+ssh://lnxscrpt@linux/usr/local/scripts
Checked out revision 0.

We will have ~/work/scripts directory now

After we check out, other people may have made some changes to the repository, it's always a good practice to update our local copy before changing anything.
$ svn up scripts
At revision 0.

2. To add check_load.sh, copy check_load.sh to ~/work/scripts, and run
$  svn add scripts/check_load.sh
A         scripts/check_load.sh

3. Commit our changes:
$ svn ci scripts/
Adding         scripts/check_load.sh
Transmitting file data .
Committed revision 1.

4. Make sure check_load.sh is in SVN repository:
$  svn ls svn+ssh://lnxscrpt@linux/usr/local/scripts
check_load.sh
check_mem.sh

5. check svn log
$ svn log svn+ssh://lnxscrpt@linux/usr/local/scripts/check_load.sh
------------------------------------------------------------------------
r1 | lnxscrpt | 2013-09-03 14:25:11 +0800 (Tue, 03 Sep 2013) | 2 lines

added check_load.sh at 2:25pm

------------------------------------------------------------------------

$ svn log svn+ssh://lnxscrpt@linux/usr/local/scripts/check_mem.sh
------------------------------------------------------------------------
r5 | lnxscrpt | 2013-09-03 14:44:38 +0800 (Tue, 03 Sep 2013) | 2 lines

import check_mem.sh

------------------------------------------------------------------------

Note: whenever we svn commands, we will be prompted the password for lnxscrpt@linux, we can set up password ssh login by following: http://linuxscripter.blogspot.com/2012/05/set-up-password-less-ssh-login-using.html

Monday 2 September 2013

Using Nagios and NRPE to monitor remote hosts

We have setup Nagios to monitor our localhost centos, but how to monitor remote hosts? We can monitor remote hosts using SNMP or NRPE
In our testing environment, we will setup Nagios to monitor host centos-1 and centos-2 using NRPE.
To use NRPE, we need to install NRPE on both monitoring host and remote hosts:

Steps to setup remote hosts for NRPE monitoring:
1. Download nagios-plugins-1.4.16.tar.gz, nrpe-2.14.tar.gz
2. Create account for NRPE service
# useradd nagios
# passwd nagios

3. Install nagios-plugs
# tar zxpf nagios-plugins-1.4.16.tar.gz
# cd nagios-plugins-1.4.16
# ./configure --with-nagios-user=nagios \
  --with-nagios-group=nagios

# make
# make install
# chown nagios:nagios /usr/local/nagios

4. Install NRPE
# tar zxpf nrpe-2.14.tar.gz
# cd nrpe-2.14
# ./configure
# make all
# make install-plugin
# make install-daemon
# make install-daemon-config

5. Modify /usr/local/nagios/etc/nrpe.cfg to allow host centos(192.168.100.100) to connect to NRPE
allowed_hosts=127.0.0.1,192.168.100.100

6. Start NRPE, enable autostart to NRPE
# /usr/local/nagios/bin/nrpe -c \
  /usr/local/nagios/etc/nrpe.cfg -d

# echo '/usr/local/nagios/bin/nrpe -c \
  /usr/local/nagios/etc/nrpe.cfg -d' >> /etc/rc.local

7. Verify NRPE is working properly
# /usr/local/nagios/libexec/check_nrpe -H localhost
NRPE v2.14

Steps to setup monitoring host for NRPE monitoring
1. Download nrpe-2.14.tar.gz
2. Install NRPE
# tar -zxpf nrpe-2.14.tar.gz
# cd nrpe-2.14
# ./configure
# make all
# make install-plugin

3. Enable monitoring of centos-1
Update /usr/local/nagios/etc/objects/commands.cfg to enable command check_nrpe
# ‘check_nrpe’ command definition
define command{
        command_name check_nrpe
        command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -t 30 -c $ARG1$
}

update /usr/local/nagios/etc/nagios.cfg to include this line
cfg_file=/usr/local/nagios/etc/objects/centos-1.cfg

4. Edit /usr/local/nagios/etc/objects/centos-1.cfg
define host{
        use                 linux-server
        host_name           centos-1
        alias               centos-1
        address             192.168.100.101
        }


define hostgroup{
        hostgroup_name      centos
        alias               centos
        members             centos-1
        }


define service{
        use                 generic-service
        host_name           centos-1
        service_description PING
        check_command       check_ping!100.0,20%!500.0,60%
        }

define service{
        use                 generic-service
        host_name           centos-1
        service_description Server Load
        check_command       check_nrpe!check_load
        }

define service{
        use                 generic-service
        host_name           centos-1
        service_description /rhiso Usage
        check_command       check_nrpe!check_rhiso
        }

5. Restart Nagios service
# service nagios restart

Steps to enable nrpe commands check_load and check_rhiso on remote host centos-1
Edit /usr/local/nagios/etc/nrpe.cfg, add these 2 lines:
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
command[check_rhiso]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% /rhiso

restart nrpe daemon
# pkill nrpe
# /usr/local/nagios/bin/nrpe -c \
  /usr/local/nagios/etc/nrpe.cfg -d

Login to http://centos/nagios again, you will see now we have host centos-1 monitored, similarly we can setup the monitoring of centos-2

Setting up Nagios monitoring server

Nagios is an open-source monitoring software, you can monitor different host resources like host UP/down status, disk usage, system load, logfile.
Setting up Nagios on Linux is very straight forward.

In our test environment, we will setup host centos as our monitoring host.

Steps to install Nagios on centos:
1. Download nagios core, nagios plugins
http://www.nagios.org/download/core, choose DIY option.
http://www.nagios.org/download/plugins, download latest stable version.

2. Create nagios account:
# useradd nagios
# passwd nagios
# groupadd nagcmd
# usermod -a -G nagcmd nagios
# usermod -a -G nagcmd apache

3. Install Nagios
# tar -zxpf nagios-3.5.0.tar.gz
# cd nagios
# ./configure --with-command-group=nagcmd
# make all
# make install
# make install-init
# make install-config
# make install-commandmode
# make install-webconf

4. Create nagiosadmin account, this account will be used to login to nagios web interface
# htpasswd -c /usr/local/nagios/etc/htpasswd.users \
  nagiosadmin


5. Edit /etc/httpd/conf.d/nagios.conf, pointing AuthUserFile to /usr/local/nagios/etc/htpasswd.users

6. Start apache
# service httpd restart

7. Install Nagios Plugins
# tar -zxpf nagios-plugins-1.4.16.tar.gz
# cd nagios-plugins-1.4.16
# ./configure --with-nagios-user=nagios \
  --with-nagios-group=nagios

# make; make install

8. Enable autostart of Nagios and start Nagios service
# chkconfig --add nagios
# chkconfig nagios on

# service nagios start
9. Login to http://centos/nagios, you will see we already have centos monitored by Nagios


In the next post http://linuxscripter.blogspot.com/2013/09/using-nagios-and-nrpe-to-monitor-remote.html, I will discuss how to monitor remote hosts using Nagios and NRPE.

Reference: http://nagios.sourceforge.net/docs/3_0/quickstart-fedora.html