setup nginx web server with PHP

Nginx (engine x) is a high performance lightweight HTTP server, more and more sites are using nginx, according to Netcraft survey (http://news.netcraft.com/archives/2013/11/01/november-2013-web-server-survey.html), nginx powers 15% of the busies sites in November 2013.

Nginx installation is very straight forward, we can download latest source code from http://nginx.org/en/download.html or point our yum source to http://nginx.org/packages/OS/OSRELEASE/$basearch/ and install using yum.
Replace “OS” with “rhel” or “centos”, depending on the distribution used, and “OSRELEASE” with “5” or “6”, for 5.x or 6.x versions, respectively.
So for CentOS 6.3, we can point our YUM source to: http://nginx.org/packages/centos/6/$basearch/

How to recover deleted open files

In Linux, a file is deleted completed when:

1. No more hard link reference to the file
2. Processes opening the file is terminated

From why du and df show different filesystem usage (http://linuxscripter.blogspot.com/2013/11/why-du-and-df-show-different-filesystem.html), we know that if we delete an open file, Linux won't release its space until the opening processes are stopped.

So if we delete an open file by mistake, is there a way to recover it?
YES, we can check which process is opening the file, and recover file content by checking the process file descriptor.

Again, let's assume our Apache error_log is deleted, we can check which process is opening this file:

[root@centos ~]# lsof | sed -n '1p;/error_log.*deleted/p'
COMMAND    PID      USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME
httpd     3155      root    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3157    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3158    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3159    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3160    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3161    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3162    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3163    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3164    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)

From the output we know that, process 3155 still opens this file. Go to /proc/3155/fd/ to confirm this, 2w means error_log is opened for write, and we need to check softlink "2" in /proc/3155/fd.

[root@centos ~]# cd /proc/3155/fd/
[root@centos fd]# ls -l 2
l-wx------ 1 root root 64 Nov  5 09:46 2 -> /var/log/httpd/error_log (deleted)
[root@centos fd]# tail 2
[Tue Nov 05 09:46:35 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Nov 05 09:46:35 2013] [notice] Digest: generating secret for digest authentication ...
[Tue Nov 05 09:46:35 2013] [notice] Digest: done
[Tue Nov 05 09:46:35 2013] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_wsgi/3.2 Python/2.6.6 configured -- resuming normal operations

To recover the content of error_log, we can just copy 2 to a temporary location, stop Apache and rename copy of 2 to /var/log/httpd/error_log

[root@centos fd]# cp 2 /tmp/error_log
[root@centos ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@centos ~]# mv /tmp/error_log /var/log/httpd/error_log
[root@centos ~]# service httpd start
Starting httpd:                                            [  OK  ]

How to setup MySQL replication

MySQL replication is the technique to replicate one mysql database (master) to one or more databases (slave), using replication, we can:

1) Separate read and write
Write is only done on master server, all the read operations are done one the slave servers. We can also run expensive report on slave servers without affecting the performance of master and other slaves.

2) backup.
MySQL stores all the files used for database/tables in data/ directory, to backup mysql, we can backup the data/ directory. However backing up data/ while MySQL is running will give us an inconsistent copy of data/. To make consistent backup, we need to stop MySQL first, and this will cause downtime.
In MySQL replication environment, we can stop one slave and do the backup on slaves. Master and other slaves are still online during our backup.


To setup MySQL replication, each server in the replication needs to have a unique server-id. The valid range of server-id is 0 to 2^32-1, but replication only allows positive server-id.

I have 3 servers called: centos, centos-1, centos-2. All servers have MySQL freshly installed.

[root@centos ~]# getent hosts | grep centos
192.168.100.100 centos
192.168.100.101 centos-1
192.168.100.102 centos-2

Below are the steps I did to setup replication from centos to centos-1 and centos-2.
1. update /etc/my.cnf and restart MySQL
on centos:

[root@centos ~]# cat <<EOF > /etc/my.cnf
[mysqld]
server-id = 100
log-bin = master-bin.log 
EOF 
[root@centos ~]# service mysql restart

on centos-1:

[root@centos-1 ~]# cat <<EOF > /etc/my.cnf
[mysqld]
server-id = 101
relay-log-index = slave-relay-bin.index
relay-log = slave-relay-bin
replicate-wild-ignore-table=mysql.%
replicate-wild-ignore-table=information_schema.%
replicate-wild-ignore-table=performance_schema.%
EOF
[root@centos-1 ~]# service mysql restart

on centos-2:

[root@centos-2 ~]# cat <<EOF > /etc/my.cnf
[mysqld]
server-id = 102
relay-log-index = slave-relay-bin.index
relay-log = slave-relay-bin
replicate-wild-ignore-table=mysql.%
replicate-wild-ignore-table=information_schema.%
replicate-wild-ignore-table=performance_schema.%
EOF
[root@centos-2 ~]# service mysql restart

2. Get master bin log and position.
mysql> flush table with read lock;
Query OK, 0 rows affected (0.00 sec)

mysql> show master status;

+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| master-bin.000007 |      272 |              |                  |
+-------------------+----------+--------------+------------------+

1 row in set (0.00 sec)

mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)

3. On master, create account used for replication. Slave uses this account to connect to master. Multiple clients can share the same account, we can also create one account for each slave. This account need to have privilege "REPLICATION SLAVE".

mysql> CREATE USER 'repl'@'%' IDENTIFIED BY 'slavepass';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';
mysql> FLUSH PRIVILEGES;

4. Setup slave using the information found in step 2 and 3.
On both centos-1 and centos-2

mysql> change master to
    -> master_host='192.168.100.100',
    -> master_port=3306,
    -> master_user='rep1',
    -> master_password='slavepass',
    -> master_log_file='master-bin.000007',
    -> master_log_pos=272;
mysql> start slave;

5. Confirm replication is working:

[root@centos-1 data]# tail -1 centos-1.err
130626  9:47:16 [Note] Slave I/O thread: connected to master 'rep1@192.168.100.100:3306',replication started in log 'master-bin.000007' at position 272
[root@centos-2 data]# tail -1 centos-2.err
130626  9:48:04 [Note] Slave I/O thread: connected to master 'rep1@192.168.100.100:3306',replication started in log 'master-bin.000007' at position 272

we can also insert some data on master, querying on any slave will show us the data we inserted on master.

We are done! MySQL replication is up and running.

reference:
http://dev.mysql.com/doc/refman/5.6/en/replication-howto.html
http://top-performance.blogspot.com/2012/03/how-to-setup-mysql-replication-in-11.html

How to check NIC speed in Unix

Fast Ethernet connection is essential for some operations to work properly.
backup, oracle RAC all require fast Ethernet connection.

Different OS provides different commands to check the network speed currently running.

In Linux, we can run mii-tool or ethtool to check the network speed. Both commands require root access

[root@localhost eth0]# mii-tool eth0
eth0: no autonegotiation, 100baseTx-FD, link ok

[root@localhost eth0]# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: umbg
        Wake-on: d
        Current message level: 0x00000007 (7)
        Link detected: yes

In Solaris, we can run kstat or dladm to check the network speed

$ kstat -m igb -i 0 | egrep 'link_autoneg|link_speed|link_duplex'
        link_autoneg                    1
        link_duplex                     2
        link_speed                      100

The output shows that, igb0 is running autonegotiation, 100 Mbps full-duplex

if we have root access, we can run dladm or ndd to check the network speed.

In AIX, we can run entstat to check network speed.
if we want to check the speed of en0

$ entstat -d en0 | egrep '^Device|^Media'
Device Type: 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
Media Speed Selected: Auto negotiation
Media Speed Running: 100 Mbps Full Duplex

We can see that en0 supports upto 1000 Mbps, but it's running 100 Mbps Full Duplex.

In HP-UX, we can run lanadmin or nwmgr to check the network speed.
lanadmin output is shorter

$ /usr/sbin/lanadmin -x 0
Speed = 1000 Full-Duplex.
Autonegotiation = On.

we can also use nwmgr to get detailed information

$ /usr/sbin/nwmgr --get --stats -C lan -I 0

***          lan0 64 bit MIB statistics:
Interface Name               = lan0
PPA Number                   = 0
Description                  = lan0 HP PCI 1000Base-T Release B.11.31.0809.01
Interface Type               = 1000Base-T
MTU Size                     = 1500
Speed                        = 1 Gbps
Station Address              = 0x0A527E583C90
Administration Status        = UP
Operation Status             = UP
Last Change                  = Wed Mar 20 05:58:50 2013
Inbound Octets               = 166627219743
Inbound Unicast Packets      = 975967141
Inbound Multicast Packets    = 1124454
Inbound Broadcast Packets    = 6566058
Inbound Discards             = 1821280
Inbound Errors               = 0
Inbound Unknown Protocols    = 1124769
Outbound Octets              = 185484791114
Outbound Unicast Packets     = 968101273
Outbound Multicast Packets   = 0
Outbound Broadcast Packets   = 1618
Outbound Discards            = 0
Outbound Errors              = 0
Counter Discontinuity Time   = Wed Mar 20 05:58:50 2013
Physical Promiscuous Mode    = FALSE
Physical Connector Present   = TRUE
Interface Alias              =
Link Up/Down Trap Enable     = Enabled

Redhat Linux: how to add/remove hard disk without rebooting server

A few years ago, I managed a few hundred of Redhat Linux servers, most of them are DELL PowerEdge 1750, 1850, and 1950.
Sometimes I needed to move the harddisk between servers. Removing or adding hardisks normally requires system reboot.
But I used this way to remove or add disk without rebooting:

echo 'scsi remove-single-device 0 0 1 0' > /proc/scsi/scsi
echo 'scsi add-single-device 0 0 1 0' > /proc/scsi/scsi

Recently I was asked to add a new harddisk without reboot system. It's a Redhat in VMware, as I was not managing VMware, so not clear about the hostadaptor, SCSI channel, ID, LUN, manipulating /proc/scsi/scsi may not work.

After searching around I found that I can let system to rescan the SCSI bus.

for scsi_host in /sys/class/scsi_host/host*
do
    echo '- - -' > $scsi_host/scan
done

After this, I could see the new hard disk was spinned up in /var/log/message.

Besides 'scsi remove-single-device 0 0 1 0', we can also do it in this way to remove a hard disk.

echo 1 > /sys/block/sdb/device/delete

Reference:
http://www.tldp.org/HOWTO/archived/SCSI-Programming-HOWTO/SCSI-Programming-HOWTO-4.html
http://www.cyberciti.biz/tips/vmware-add-a-new-hard-disk-without-rebooting-guest.html

BIND DNS: reverse delegation of IP range

Besides mapping domain name to IP address, DNS systems can also map IP address to domain name. Many applications rely on DNS reverse mapping to function properly.
NetBackup will throw error if reverse mapping is not setup or is setup incorrectly, email system also needs correct reverse mapping.

Suppose we have the whole range of IP addresses in 222.222.222.0/24, we can configure reverse mapping in named.conf:

zone "222.222.222.in-addr.arpa" {

    type master;

    file "222.222.222.rev";

};

If we only have part of the IP addresses in 222.222.222.0/24, suppose 222.222.222.64 to 222.222.222.91, the reverse mapping is called classless reverse delegation.

From 222.222.222.64 to 222.222.222.95 there are 30 usable addresses, plus the network and broadcast address, there are 32 addresses, 32 = 2^5, 8 x 4 - 5 = 27. so our IP range can be represented as 222.222.222.64/27

ISP should have defined reverse delegation in their reverse zone file:

64/27    IN    NS    ns.sg.linuxscripter.blogspot.com

Now we can define the reverse mapping for our IP range in our own named.conf:

zone "64/27.222.222.222.in-addr.arpa" {

    type master;

    file "64-95.222.222.222.rev";

};

Note: The domain name and IP addresses in this post are dummy ones, I use them for easier writing.

BIND DNS: subdomain delegation

Suppose we have the domain name linuxscripter.blogspot.com, and we have Singapore, Hong Kong, and Shanghai offices. If we want the subsidiaries to manage their own domains, we can do this using domain delegation. On headquarter DNS server, in linuxscripter.blogspot.com zone file, we can define the subdomain delegation for different subsidiaries.

$ORIGIN sg.linuxscripter.blogspot.com.
@       IN       NS       ns.sg.linuxscripter.blogspot.com.
ns      IN       A        221.221.221.221

$ORIGIN hk.linuxscripter.blogspot.com.
@       IN       NS       ns.hk.linuxscripter.blogspot.com.
ns      IN       A        222.222.222.222

$ORIGIN sh.linuxscripter.blogspot.com.
@       IN       NS       ns.sh.linuxscripter.blogspot.com.
ns      IN       A        223.223.223.223

In ns.sg.linuxscripter.blogspot.com, we can define zone file for subdomain sg.linuxscripter.blogspot.com as normal domains

$TTL 7d

$ORIGIN sg.linuxscripter.blogspot.com.

@              IN      SOA   ns.sg.linuxscripter.blogspot.com. hostmaster.abc.domain. (
               2013022701 ; serial number
               2h         ; refresh =  2 hours
               15M        ; update retry = 15 mins
               3W12h      ; expiry = 3 weeks + 12 hours
               2h20M      ; minimum = 2 hours + 20 mins
               )

      IN   NS     ns.sg.linuxscripter.blogspot.com.
      IN   MX     10 mail.sg.linuxscripter.blogspot.com.

www   IN   A      221.221.221.65
mail  IN   A      221.221.221.66

We can setup the web server to use www now, but to use the email server, we need to define the reverse resolution for mail.sg.linuxscripter.blogspot.com properly.

Depends on the IP address range we get from ISP, the syntax for reverse resolution may involve classless delegation, you can find details on how to do reverse delegation in http://linuxscripter.blogspot.sg/2013/03/bind-dns-reverse-delegation-of-ip-range.html

Note: The domain name and IP addresses in this post are dummy ones, I use them for easier writing.

How to get the process listening on certain port

At times we are asked: "what program is listening on port XX?"

In Redhat, we can easily get this using lsof. let's say if we want to know which program is listening on port 80.
[root@localhost ~]# lsof -i :80
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
httpd   2047   root    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2049 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2050 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2051 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2052 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2053 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2054 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2055 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2056 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)

From the output, we know that httpd is listening on port 80.

Another way to get the process listening on certain port is using netstat.
[root@localhost ~]# netstat -anp | grep 80
tcp        0      0 :::80     :::*            LISTEN      2009/httpd
 
[root@localhost ~]# ps -ef | awk '$2 == 2009'
root      2009     1  0 23:21 ?        00:00:00 /usr/sbin/httpd
 
[root@localhost ~]#
In AIX, most of the time it has no lsof installed, and the netstat is also different from Redhat, but we still can get our question answered.
$ netstat -Aan | grep '.22 ' | grep LISTEN
f1000e0001382bb8 tcp4   0  0  *.22          *.*        LISTEN

$ rmsock f1000e0001382bb8 tcpcb
The socket 0xf1000e0001382808 is being held by proccess 3670142 (sshd).
 
$ ps -ef | grep 3670142
root 3670142 3014676   0   Sep 06      -  0:00 /usr/sbin/sshd

Please note rmsock will not remove the socket, you can confirm this by checking the content of /var/adm/ras/rmsock.log.

reference: http://unix.ittoolbox.com/groups/technical-functional/ibm-aix-l/determine-which-process-is-listening-on-a-port-without-using-lsof-on-aix-1468555

How to setup NFS in RedHat and Solaris

NFS stands for Network File System, we can use it to share files between different computers.

In Redhat Linux, to share NFS, we need to configure /etc/exports, the format of exports is:

<directory to share>   <host>(<option>)

For example, if we want to share directory /data/forhost1 to host1, we can configure my /etc/exports like this:

/data/forhost1      host1

if we only want to grant read permission to /data/forhost1, we can add the ro option, ro stands for read-only

/data/forhost1     host1(ro)

Solaris uses a different configuration file /etc/dfs/dfstab for NFS.
If we want to share directory /data/forhost1 to host1, we can configure my /etc/dfs/dfstab like this:

share -F nfs -o ro=host1 -d "Readonly" /data/forhost1

For both RedHat Linux and Solaris, we need to restart the nfs service to make our configuration take effect, this can be done by using service or svcadm.

After we share /data/forhost1, we can mount it on host1:
if host1 is Redhat Linux

mount -t nfs server:/data/forhost1 /mnt/mynfs

if host1 is Solaris:

mount -F nfs server:/data/forhost1 /mnt/mynfs

To make the nfs mount permanent, we can configure /etc/fstab or /etc/vfstab to mount nfs on system start.

How to get hardware information in Linux

We often need to get the hardware information, In Redhat Linux, this can be easily done by using the dmidecode.
To get the bios information:

# dmidecode -t 0

In dmidecode, 0 is the type code for BIOS.

By using dmidecode, we can get many information about the system, for details, please refer to the man page

# man dmidecode

All the Linux systems I have been managed are Redhat running on Dell servers, so I am also using Dell OpenManage to get the system hardware information.
To get the bios information

$ omreport chassis bios

There are also many other options we can use in Dell OpenManage, for details, please refer to: http://support.dell.com/support/edocs/software/svradmin/5.1/en/omss_ug/html/cli.html#1220531

Please note that we can only run dmidecode as super user, whereas we can run omreport as normal user.

Linux: how to recover root password

In Linux, if you forget the root password, you can recover it in this way:

1. Reboot the machine, at grub menu, edit the boot entry for Linux
2. press 'e' to to edit the entry,
   append " single" at the end of the kernel line
   alternatively, you can also append " init=/bin/sh" at the end of the kernel line
3. press 'b' to boot Linux into single user mode
4. now you will be able to log in the system without entering password 
5. type 'passwd' to reset root password

Set up password-less SSH login using ssh-keygen and ssh-copy-id

Logging in remote systems without password is very useful in system administration. It can be used for executing command on remote systems, copying files to other systems using rsync/rdist.

For the past few years, I have been manually setting up the key trust between systems. Here are the steps I followed.

Suppose I need to login to localserver from remoteserver using account scripter without password.

1. generate public-private key pair on localserver

$ ssh-keygen -t dsa

    you can use either dsa or rsa.

2. copy scripter's public key to remoteserver

$ scp id_dsa.pub remoteserver:

3. on remoteserver, append the public key copied in step 2 to ~scripter/.ssh/authorized_keys

$ cat id_dsa.pub >> ~scripter/.ssh/authorized_keys

    if ~scripter/.ssh doesn't exist, manually create it

4. on localserver the directory and files are auto-generated, so the permissions are correct, on remoteserver, as we manually created the directories, we need to make the permissions are correct.

$ chmod 700 ~scripter/.ssh

$ chmod 600 ~scripter/.ssh/authorized_keys

After all these steps, we can login from localserver to remoteserver without entering password:

$ ssh remoteserver hostname

remoteserver

The steps are quite easy, but can it be easier?
Yes! recently I learned the script ssh-copy-id from a forum post, to set up key trust from localserver to remoteserver using ssh-keygen and ssh-copy-id
1. generate public-private key pair on localserver

$ ssh-keygen -t dsa

2. copy the public key from localserver to remoteserver, and set up the key trust

$ ssh-copy-id -i ~scripter/.ssh/id_dsa.pub \
scripter@remoteserver

After you entering the password, everything is set up for you, you can log in remotely without password

$ ssh remoteserver hostname

remoteserver

ssh-copy-id saves us all the troubles of creating files and setting up the proper permission :)

Copy files from one server to another

In my work, I often need to copy files or whole directories between different servers. These are the ways I used often:

1. scp

scp -r directory remote:/path/to/directory

2. tar + ssh

tar -zcpf - directory | ssh remote tar -zxpf \
-C /path/to/directory

in Solaris, there is no -C -z flag, so I use:

tar -cpf - directory | ssh remote \
"(cd /path/to/directory; tar -xpf -)"

3. rsync over ssh
I have a central server, everyday there are some new files generated in one directory. I need to push this directory to remote servers everyday.
It's possible to schedule some cronjob to scp or tar + ssh this directory, But it will copy the whole directory everyday. To use rsync, we only need to copy the changes to this directory.

rync -e ssh -az --delete directory \
remote:/path/to/directory

4. nc + tar
A few years ago, I need to copy huge amount of data to a remote data center in another country, while exploring ways to copy the data, I found this utility nc.
But nc doesn't encrypt the data, I only tested it within the same data center.

on remote server:

cd /path/to/directory

nc -l -p 8888 | tar -zxvf -

on source server:

tar -czf - directory | nc remote 8888

on the remote server, I used -v flag, so I could monitor the copying progress.

Understanding Linux Hugepages

Hugepages can be extremely useful for systems having bigger RAM.

$ grep Huge /proc/meminfo

HugePages_Total:         0

HugePages_Free:          0

HugePages_Rsvd:          0

HugePages_Surp:          0

Hugepagesize:         2048 kB

Hugepagesize is the size of one Hugepage, on most x86 or x86_64 machines,  it's set to 2048 kB.
HugePages_Total is the number of Hugepages configured on the machine. In Redhat Linux, we can change this value by updating /etc/sysctl.conf

echo "vm.nr_hugepages = 200" >> /etc/sysctl.conf

 After updating, /sbin/sysctl -p may make the setting take effect, but it's better to restart the machine.

To confirm Hugepages is configured after rebooting,

$ grep Huge /proc/meminfo

HugePages_Total:       200

HugePages_Free:        200

HugePages_Rsvd:          0

HugePages_Surp:          0

Hugepagesize:         2048 kB

By setting Hugepages to 200, we are telling linux to reserve 200 Hugepages. So the memory associated with Hugepages is: HugePages_Total x Huagepagesize = 200 x 2048kB = 400M.
The memory associated with Hugepages is always allocated, and it cannot be swapped out.

Before setting Hugepages,

$ grep Mem /proc/meminfo

MemTotal:                1030940 kB

MemFree:                  861852 kB

After setting Hugepages, check memory usage again:

$ grep Mem /proc/meminfo

MemTotal:                1030940 kB

MemFree:                  452252 kB

We can see that the free memory dropped from 841M to 441M, after setting HugePages_Total to 200. The 400M memory is the associated to Hugepages, and it will always be allocated.

We can use Hugepages to lock some memory for oracle SGA, so the SGA will never be paged out, thus improving the database performance.

Use PAM to enforce Linux password complexity

It's always an audit requirement to have a system not too short and not so easy to be guessed.
To enforce the password length, we can use /etc/login.defs

PASS_MIN_LEN      8

next time when user changes password, anything shorter than 8 characters will be rejected.

To enforce the password complexity, we have to make sure it consists of  uppercase, lowercase, special characters, and digits. This can be easily done through the use of PAM.

$ man pam_cracklib

lcredit=N

(N >= 0) This is the maximum credit for having lower case letters in the new password.
(N < 0) This is the minimum number of lower case letters that must be met for a new password.

So to force at least 1 lowercase character in the password, we should use negative number, lcredit=-1
To enforce a password having at least 4 lower cases, 2 upper cases, 1 special character, and 1 digit, we can update the /etc/pam.d/system-auth

password requisite pam_cracklib.so dcredit=-1 ucredit=-2 lcredit=-4 ocredit=-1

Remove unused filesystem from linux

On one of the redhat server, found a non-standard filesystem was mounted under /migwork.
From the name, it looks like it was used for some migration work.

check whether the file system is still in use

$ fuser /migwork

No process is using /mitwork, so we can safely remove it.

$ umount /migwork

update /etc/fstab to prevent it from being mounted after reboot.

#/dev/md20        /migwork        ext3        defaults     1 2

check the information about md20

$ cat /proc/mdstat
$ mdadm --detail /dev/md20 | sed -n '/RaidDevice/,$p'

destroy md20

$ mdadm --stop /dev/md20
$ mdadm --remove /dev/md20

mdstat showed these sub disks are from SAN, inform storage engineer to take them back.

How to disable fsck on reboot

In Linux, when a filesystem is mounted for certain times, or its last fsck was more than certain days ago, system will perform fsck on it when server reboot. The fsck process can take a few minutes to hours to finish, depending on the filesystem size.

If we want fast reboot, we can disable the fsck check, although it's not recommended to do so.
To disable fsck, there are a few ways.

1. update /etc/fstab
in /etc/fstab, the last column is used by fsck to determine the order of performing file system check at reboot time. For root file system /, it should be 1, for other file systems, it should be 2. If we want to disable the fsck check for certain file system, we can specify 0 in the last column.

$ grep nofsck /etc/fstab
/dev/sda2        /mnt/nofsck        ext4        defaults        0  0

2. use tune2fs to change the filesystem parameters
To list the current settings:

$ tune2fs /dev/sda2 | egrep -i 'mount count|check'
Mount count:                        1
Maximum mount count:        21
Last checked:                     Sat Mar 24 16:15:33 2012
Check interval:                    15552000 (6 months)
Next check after:                 Thu Sep 20 16:15:33 2012

the output is self-explained, for my system, /dev/sda2 will be checked after it's mounted for 21 times, or after Sep 20 16:15:33 2012.

To disable fsck check on /dev/sda2

$ tune2fs -c 0 -i 0 /dev/sda2
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to -1
Setting interval between checks to 0 seconds.

check it again:

$ tune2fs /dev/sda2 | egrep -i 'mount count|check'
Mount count:                        1
Maximum mount count:        -1
Last checked:                     Sat Mar 24 16:15:33 2012
Check interval:                    0 (<none>)

3. Tell system to skip fsck when rebooting.

$ /sbin/shutdown -rf now

the -f flags tells system to skip fsck for all filesystems during the reboot. Unlike the fstab and tune2fs methods, it only takes effect during current reboot, will not disable fsck permanently.

Rename multiple files using one command

On Redhat Linux machine, suppose we have hundreds of .txt files, if want want to rename them to .sh file, there are a few ways to do:

1. use rename command

rename txt sh *.txt

2. write script

for f in *.txt; do mv $f ${f/.txt/.sh}; done

${f/.txt/.sh} is string replacement in bash, for details, please see string manipulation in bash