Sunday, 26 May 2013

bash string manipulation

Bash supports many ways of string manipulation, it includes substring, substring replacement, substring removal, string length.

In how to rename multiple files in one command, one of the ways I shared to rename *.txt to *.sh is

for f in *.txt
do
        mv $f ${f/.txt/.sh}
done

How ${f/.txt/.sh} is doing the magic? Actually this is string replacement in bash.

${f/.txt/.sh} means replacing ".txt" in string $f with ".sh", so for f=abc.txt, ${f/.txt/.sh} will produce abc.sh

[linuxscripter@localhost ~]$ f=abc.txt
[linuxscripter@localhost ~]$ echo $f{f/.txt/.sh}
abc.sh

But the example I gave in rename multiple files in one command was not so correct.
For f=abc.txt.txt, ${f/.txt/.sh} will produce abc.sh.txt instead of abc.txt.sh

[linuxscripter@localhost ~]$ f=abc.txt.txt
[linuxscripter@localhost ~]$ echo $f{f/.txt/.sh}
abc.txt.sh

To correct this, we need to use ${f/%.txt/.sh}

[linuxscripter@localhost ~]$ f=abc.txt.txt
[linuxscripter@localhost ~]$ echo $f{f/%.txt/.sh}
abc.txt.sh

The syntax of string replacement includes:
1. ${var/Pattern/Replacement}   : replacing the first match of "Pattern" with "Replacement"
2. ${var//Pattern/Replacement}  : replacing every match of "Pattern" with "Replacement"
3. ${var/#Pattern/Replacement}  : if $var starts with "Pattern", replace "Pattern" with "Replacement"
4. ${var/%Pattern/Replacement}  : if $var ends with "Pattern", replace "Pattern" with "Replacement"

Some examples will make the syntax clearer.

[linuxscripter@localhost ~]$ f=abc.txt.abc.txt

[linuxscripter@localhost ~]$ ### testing syntax 1
[linuxscripter@localhost ~]$ echo ${f/txt/sh}
abc.sh.abc.txt

[linuxscripter@localhost ~]$ ### testing syntax 2
[linuxscripter@localhost ~]$ echo ${f//txt/sh}
abc.sh.abc.sh

[linuxscripter@localhost ~]$ ### testing syntax 3, [linuxscripter@localhost ~]$ echo ${f/#txt/sh}
abc.txt.abc.txt

[linuxscripter@localhost ~]$ ### testing syntax 3,

[linuxscripter@localhost ~]$ echo ${f/#abc/sh}
sh.txt.abc.txt

[linuxscripter@localhost ~]$ ### testing syntax 4,
[linuxscripter@localhost ~]$ echo ${f/%abc/sh}
abc.txt.abc.txt
[linuxscripter@localhost ~]$ ### testing syntax 4
[linuxscripter@localhost ~]$ echo ${f/%txt/sh}
abc.txt.abc.sh

At first these syntax may not appear very intuitive, I found this method help to understand and memorize them.
Variable is referred by add '$' infront of it
1. '/' is used for replacement, same as what we do in awk and sed
2. Once we know '/' is for replacement, '//' should be too difficult for us.
3. On keyboard, '#' is at the left of '$', so '#' is used to replace the left side of the variable.
4. On keyboard, '%' is at the right of '$', so '%' is used to replace the right side of the variable.

Besides string replacement, of course there are a lot more bash can do.
${#var} returns length of $var
[linuxscripter@localhost ~]$ echo {#f}
15

${var:pos:len} returns substring of $var, starting at position "pos", with length upto "len", if len is empty or len is too big, return substring starting at position "pos" until to the end of the string.
[linuxscripter@localhost ~]$ echo ${f:2:5} ${f:2:100} ${f:2}
c.txt c.txt.abc.txt c.txt.abc.txt

${var#substring} delete shortest match of "substring" from front of $var
[linuxscripter@localhost ~]$ echo ${f#abc}
.txt.abc.txt
[linuxscripter@localhost ~]$ echo ${f#*txt}
.abc.txt

${var##substring} delete longest match of "substring" from front of $var
[linuxscripter@localhost ~]$ echo ${f##*abc}
.txt

${var%substring} delete longest match of "substring" from back of $var
[linuxscripter@localhost ~]$ echo ${f%txt}
abc.txt.abc.
[linuxscripter@localhost ~]$ echo ${f%abc*}
abc.txt.

${var%%substring} delete longest match of "substring" from back of $var
[linuxscripter@localhost ~]$ echo ${f%%txt*}
abc.

http://tldp.org/LDP/abs/html/string-manipulation.html
http://bbs.chinaunix.net/forum.php?mod=viewthread&tid=218853&page=7

Friday, 17 May 2013

Use HTML pre tag to keep email display format

In my servers, I have some shell scripts to send system status reports to my email address.
It's very easy to implement, just run script to generage the report and send the report using sendmail, mail, or mailx.

e.g. if I need to email the output of w, I can simply put it in this way:
w | mail linuxscripter@myemail.com

but it doesn't have any subject, To fields.

Personally I prefer sendmail, so I added the email headers in this way.
(cat <<EOF
From: linuxscripter@myserver.com
To: linuxscripter@myemail.com
Subject: output of w

EOF
w)| sendmail -t

This looked pretty ok, and the email displayed nicely in my outlook express 5.

But in 2006, my company upgraded email client to outlook express 6, all the report format were gone, things should appear in the same column didn't.

How should I preserve the report format when displayed in outlook express 6?
I worked as a PHP web developer for 3 years, so the first thing came to my mind is html <pre> tag.
To use html tag in email, I need to set the email MIME type as text/html

This was how I did.
(cat <<EOF
From: linuxscripter@myserver.com
To: linuxscripter@myemail.com
Subject: output of w
Content-Type: text/html;

EOF
echo "<html><pre>"; w; echo "</pre></html>")| sendmail -t

My email reports became nicely formatted again!

Tuesday, 30 April 2013

ESXi: how to install virtual machine using VMware vSphere Client

In earliest post, I demoed How to install VMware ESXi 5.1 and vSphere Client.
This articles will show how to create and virtual machine and install centos as guest OS on ESXi host.

1. Login to VMware vSphere Client

2. Click "File -> New -> Virtual Machine .." or press ctrl-N.













3. Choose Typical configuration.









4. Specify a name for our virtual machine, here I put "vmcentos".








5.  Choose guest OS and version, I am installing centos, so choose "Linux" and "CentOS 4/5/6 (32bit)"










6. Create a disk, I am installing Centos minimal, 5G should be more than enough, leave the other option unchanged.









7. Review VM configurations before finishing, attach CD/DVD to the virtual machine. Earlier I have copied CentOS-6.3-i386-minimal.iso into ESXi using winscp, so I just attach the iso file. Under "device status" remember to check "connect at power on" , otherwise VM cannot boot up properly in step 9.














 



8. Open VM console: "Inventory -> Virtual Machine -> Open Console"











9. Press green button to boot up VM, alternatively you can click "Inventory -> Virtual Machine -> Power -> Power On" .














10. Once we see the CentOS  installation page, the following steps should be no difference from installing a physical machine.

Wednesday, 24 April 2013

How to install VMware ESXi 5.1 and vSphere client

Unlike VMware work station, Virtualbox, ESXi is a bare-metal hyper-visor, which means you don't need an operating system before you can install ESXi.

This blog article shows how I installed ESXi 5.1. The installation process is quite straight forward, and is quite similar to typical Linux installation.

There is no spare physical computer for me to test the installation, so I created a VM in my Virtualbox, from Virtualbox I installed ESXi.

1. First we need to download ESXi from VMware website, the one I downloaded is: VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso

2. Start machine with the iso downloaded in step 1.












3. After ESXi installer loaded some stuffs, we are greeted with this welcome screen and End User License Agreement (EULA).














 4.  Select the disk to install ESXi.










 5. Choose keyboard layout












6. Set root password









7. Installation started.





8. Installation completed.











9. Hit Enter to reboot ESXi, after rebooting, we are given this screen.















10. Open web browser, key in the URL given in step 9. Click "Download vSpere Client" to install vSphere Client on our windows machine.













11.  Launch vSpere Client, key in the IP adress, username, and password.


















12. After logging in, a message popup displaying "Your evaluation license will expire in 60 days", for the time being just ignore it first.
We can create or manage VMs on ESXi remotely now!












13. Once we login vSphere Client, we can create and manage our virtual machines. In http://linuxscripter.blogspot.sg/2013/04/esxi-how-to-install-virtual-machine.html, I will demo how to create VM and install guest OS.

Expired storage battery causes database performance slow

Early Monday morning, I was told our data warehouse loading process took much longer than before.
The application has been running fine for months, and there was no recent changes made on application, system, or database. The data volume is about the same as before.
AWR reports showed that the bottleneck was on system IO, looks like storage IO performance is much worse than before. So I asked the storage engineer the check storage battery status of StorageTek.

Yes, our batteres expired!

Vendor is delivering new batteries for us, replacing batteries should solve our issue.

A few years ago, when I first started as a DBA, I encountered similar issue on an OLTP database.
All of sudden, one database became very slow. I spent much more time, but still couldn't figure out what went wrong. Only after a few days, our storage engineer noticed the battery expiration in HDS.
After replacing new batteries, database performance became normal again.

Why expired batteries affect the system I/O?
Joshua Townsend has a very good article on storage basics.
In storage, there is some amount of cache RAM acting as a buffer to physical disks, I/O operations on cache is much faster than the disks.
In case of battery expiration, storage will disable the cache, all the I/O operations directly goes to the disks, which are much slower than cache RAM.

For details, please visit Joshua's blog:
http://vmtoday.com/2010/03/storage-basics-part-v-controllers-cache-and-coalescing/

Friday, 12 April 2013

How to check NIC speed in Unix

Fast Ethernet connection is essential for some operations to work properly.
backup, oracle RAC all require fast Ethernet connection.

Different OS provides different commands to check the network speed currently running.

In Linux, we can run mii-tool or ethtool to check the network speed. Both commands require root access
[root@localhost eth0]# mii-tool eth0
eth0: no autonegotiation, 100baseTx-FD, link ok
[root@localhost eth0]# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: umbg
        Wake-on: d
        Current message level: 0x00000007 (7)
        Link detected: yes

In Solaris, we can run kstat or dladm to check the network speed
$ kstat -m igb -i 0 | egrep 'link_autoneg|link_speed|link_duplex'
        link_autoneg                    1
        link_duplex                     2
        link_speed                      100
The output shows that, igb0 is running autonegotiation, 100 Mbps full-duplex

if we have root access, we can run dladm or ndd to check the network speed.

In AIX, we can run entstat to check network speed.
if we want to check the speed of en0
$ entstat -d en0 | egrep '^Device|^Media'
Device Type: 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
Media Speed Selected: Auto negotiation
Media Speed Running: 100 Mbps Full Duplex

We can see that en0 supports upto 1000 Mbps, but it's running 100 Mbps Full Duplex.

In HP-UX, we can run lanadmin or nwmgr to check the network speed.
lanadmin output is shorter
$ /usr/sbin/lanadmin -x 0
Speed = 1000 Full-Duplex.
Autonegotiation = On.

we can also use nwmgr to get detailed information
$ /usr/sbin/nwmgr --get --stats -C lan -I 0

***          lan0 64 bit MIB statistics:
Interface Name               = lan0
PPA Number                   = 0
Description                  = lan0 HP PCI 1000Base-T Release B.11.31.0809.01
Interface Type               = 1000Base-T
MTU Size                     = 1500
Speed                        = 1 Gbps
Station Address              = 0x0A527E583C90
Administration Status        = UP
Operation Status             = UP
Last Change                  = Wed Mar 20 05:58:50 2013
Inbound Octets               = 166627219743
Inbound Unicast Packets      = 975967141
Inbound Multicast Packets    = 1124454
Inbound Broadcast Packets    = 6566058
Inbound Discards             = 1821280
Inbound Errors               = 0
Inbound Unknown Protocols    = 1124769
Outbound Octets              = 185484791114
Outbound Unicast Packets     = 968101273
Outbound Multicast Packets   = 0
Outbound Broadcast Packets   = 1618
Outbound Discards            = 0
Outbound Errors              = 0
Counter Discontinuity Time   = Wed Mar 20 05:58:50 2013
Physical Promiscuous Mode    = FALSE
Physical Connector Present   = TRUE
Interface Alias              =
Link Up/Down Trap Enable     = Enabled

Monday, 18 March 2013

Redhat Linux: how to add/remove hard disk without rebooting server

A few years ago, I managed a few hundred of Redhat Linux servers, most of them are DELL PowerEdge 1750, 1850, and 1950.
Sometimes I needed to move the harddisk between servers. Removing or adding hardisks normally requires system reboot.
But I used this way to remove or add disk without rebooting:
echo 'scsi remove-single-device 0 0 1 0' > /proc/scsi/scsi
echo 'scsi add-single-device 0 0 1 0' > /proc/scsi/scsi

Recently I was asked to add a new harddisk without reboot system. It's a Redhat in VMware, as I was not managing VMware, so not clear about the hostadaptor, SCSI channel, ID, LUN, manipulating /proc/scsi/scsi may not work.

After searching around I found that I can let system to rescan the SCSI bus.
for scsi_host in /sys/class/scsi_host/host*
do
    echo '- - -' > $scsi_host/scan
done

After this, I could see the new hard disk was spinned up in /var/log/message.

Besides 'scsi remove-single-device 0 0 1 0', we can also do it in this way to remove a hard disk.
echo 1 > /sys/block/sdb/device/delete

Reference:
http://www.tldp.org/HOWTO/archived/SCSI-Programming-HOWTO/SCSI-Programming-HOWTO-4.html
http://www.cyberciti.biz/tips/vmware-add-a-new-hard-disk-without-rebooting-guest.html