Tuning TCP/IP stack in Linux

Kernel tunable parameters for TCP/IP stack performance

Tunable parameter	Default value	Option description
/proc/sys/net/core/rmem_default	"110592"	Defines the default receive window size; for a large BDP, the size should be larger.
/proc/sys/net/core/rmem_max	"110592"	Defines the maximum receive window size; for a large BDP, the size should be larger.
/proc/sys/net/core/wmem_default	"110592"	Defines the default send window size; for a large BDP, the size should be larger.
/proc/sys/net/core/wmem_max	"110592"	Defines the maximum send window size; for a large BDP, the size should be larger.
/proc/sys/net/ipv4/tcp_window_scaling	"1"	Enables window scaling as defined by RFC 1323; must be enabled to support windows larger than 64KB.
/proc/sys/net/ipv4/tcp_sack	"1"	Enables selective acknowledgment, which improves performance by selectively acknowledging packets received out of order (causing the sender to retransmit only the missing segments); should be enabled (for wide area network communication), but it can increase CPU utilization.
/proc/sys/net/ipv4/tcp_fack	"1"	Enables Forward Acknowledgment, which operates with Selective Acknowledgment (SACK) to reduce congestion; should be enabled.
/proc/sys/net/ipv4/tcp_timestamps	"1"	Enables calculation of RTT in a more accurate way (see RFC 1323) than the retransmission timeout; should be enabled for performance.
/proc/sys/net/ipv4/tcp_mem	"24576 32768 49152"	Determines how the TCP stack should behave for memory usage; each count is in memory pages (typically 4KB). The first value is the low threshold for memory usage. The second value is the threshold for a memory pressure mode to begin to apply pressure to buffer usage. The third value is the maximum threshold. At this level, packets can be dropped to reduce memory usage. Increase the count for large BDP (but remember, it's memory pages, not bytes).
/proc/sys/net/ipv4/tcp_wmem	"4096 16384 131072"	Defines per-socket memory usage for auto-tuning. The first value is the minimum number of bytes allocated for the socket's send buffer. The second value is the default (overridden by wmem_default) to which the buffer can grow under non-heavy system loads. The third value is the maximum send buffer space (overridden by wmem_max).
/proc/sys/net/ipv4/tcp_rmem	"4096 87380 174760"	Same as tcp_wmem except that it refers to receive buffers for auto-tuning.
/proc/sys/net/ipv4/tcp_low_latency	"0"	Allows the TCP/IP stack to give deference to low latency over higher throughput; should be disabled.
/proc/sys/net/ipv4/tcp_westwood	"0"	Enables a sender-side congestion control algorithm that maintains estimates of throughput and tries to optimize the overall utilization of bandwidth; should be enabled for WAN communication. This option is also useful for wireless interfaces, as packet loss may not be caused by congestion.
/proc/sys/net/ipv4/tcp_bic	"1"	Enables Binary Increase Congestion for fast long-distance networks; permits better utilization of links operating at gigabit speeds; should be enabled for WAN communication.

References

• http://www.ibm.com/developerworks/library/l-hisock/
• http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf
• http://www.ntop.org/products/pf_ring/hardware-packet-filtering

Port forwarding in Linux

To access a machine not directly accessible on network, port forwarding can be used from an accessible machine to the destination machine. 

One common scenario where port forwarding is useful is where a process, like web server, is on a machine with no x-window/browser and it is not directly accessible from the machine running the browser, but an accessible "gateway" machine is available in between.

The following can be run on the "gateway" machine:

ssh <user>@<gateway-host> -L <port>:<destination-host>:<destination-port> -g -N

As long as this process is running, gateway-host:port will show the content from <destination-host>:<destination-port>. 

Difference in interrupt counts from /proc/interrupts

The following python script can be used to determine interrupt counts occurred between the before and after "cat /proc/interrupts":

#!/usr/bin/python
'Interrupt counts between before and after /proc/interrupts snapshots'

import sys
from itertools import izip

def checkUsage():
    #Check Usage
    if len(sys.argv) < 3:
        print "Usage ->"
        print sys.argv[0]," </proc/interrupts before> </proc/interrupts after>"

def readFiles(fileName1, fileName2):
    fileA = open(fileName1)
    fileB = open(fileName2)
    for lineA, lineB in izip(fileA, fileB):
        wordsA = lineA.rstrip().split()
        wordsB = lineB.rstrip().split()
        for wordA, wordB in izip(wordsA, wordsB):
            if wordA.isdigit() and wordB.isdigit():
                sys.stdout.write("%s%s" % (str(int(wordB) - int(wordA)),"\t"))
            else:
                sys.stdout.write("%s%s" % (wordB," "))
        print

def main():
    checkUsage()
    #readFile(sys.argv[1])
    readFiles(sys.argv[1], sys.argv[2])

#Don't execute if script is imported instead of executed
if __name__ == '__main__':
    main()

Please leave me a comment if you find this post helpful.

Avoiding SSH delay

# Enable verbose mode in ssh to determine where the hang is

[local-host]$ ssh -v <remote-host>

It will result in the following:

debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to <remote-host> [<remote-IP>] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: loaded 2 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client <port1> <port2> none
debug1: kex: client->server <port1> <port2> none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '<remote-host>' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:33
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Next authentication method: password
root@<remote-host>'s password:

Sometimes the ssh appears to hang / wait for a while at the line in bold above. It is usually due to the UseDNS - default is yes. Disabling it and restarting sshd on the remote-host should fix this delay in ssh:

Add the following line in bold to /etc/ssh/sshd_config on remote-host:

#      UseDNS  Specifies whether sshd should look up the remote host name and
#      check that the resolved host name for the remote IP address maps
#      back to the very same IP address.The default is "yes".
UseDNS no

Restart sshd on the remote-host:

[remote-host]$ service sshd restart

Linux rpm commands

Following are the rpm commands I regularly use on RHEL and variants:

Command	Description	Example
rpm -qa	List all installed rpms	rpm -qa | grep kernel
rpm -qlm <rpm file> or rpm -qpl <rpm file>	List the files contained in the .rpm file
rpm -ivh <rpm file>	Check for dependency then install rpm
rpm -Uvh <rpm file>	Check for dependency then update package
rpm2cpio <rpm file> | cpio -idmv <leave empty for all or specify specific files in the rpm>	Extract files contained in the rpm (without install) into current directory

Linux: Which process is listening on a port?

Under various circumstances, like when starting up a server or a service, you may encounter "port already in use" error. Here are different ways to determine which process is using that port.

• netstat -tulpn | grep <portNumber>
• fuser <port#>/tcp
• lsof -i tcp:<port#>

To determine if the commands are available on your system, try:
whereis <command>
e.g. whereis fuser

Once the pid is known for the process using the port, get information on the running process along with its command line:
ps -aux | grep <pid>

Reference

http://www.cyberciti.biz/faq/what-process-has-open-linux-port

OutOfMemoryError: Unable to create new native thread

This post suggests how to fix OutOfMemmory (OOM) error with Java on Linux. There are 2 things to try:

[1] Reduce the stack size in JVM command line and for Linux. RHEL/Oracle Linux default is 8096k to 10240k, while even for most enterprise applications as low as 128k may suffice. HotSpot JDK7 seems to suggest a minimum of 160k. So:

• Add "-Xss160k" to the JVM commandline
• Set "ulimit -s 160" from the Linux shell and confirm with "ulimit -a". This setting is per user session. To make it persist for a user, add it to /etc/security/limits.conf

[2] Track the total number of threads on the system:

watch 'ps -efL|wc -l'

Check the system limit:

cat /proc/sys/kernel/pid_max # Usually 32768

If the total number of threads reaches this pid_max, JVM will throw OOM with unable to create new native threads error. 

To fix this issue:

• Increase the system limit as root:
  echo 65536 > /proc/sys/kernel/pid_max 
• Or, to persist across reboots, add the following to /etc/sysctl.conf:
  kernel.pid_max = 65536

Vim - Matching Braces

Matching {}, (), [], /* */, can be very helpful when using vi to read a program code.

In command mode, take your cursor to one of the start/end braces or comment mark and press %, that is, Shift + 5 on most keyboards.

Another useful trick is to highlight the code between the matching tags, and can be achieved simply by:
:noremap % v%

References

http://vim.wikia.com/wiki/Moving_to_matching_braces

Linux Point to Point Network Connection

Point to point connection is a network connection between 2 machines without a switch in between. Basically, both the ends of the cables are directly inserted into the 2 machines to be connected via p2p.

Advantages of p2p include avoiding routing overhead from the switch, simplicity, and to security.

Configuration is simply:

/sbin/ifconfig <dev> inet <local-host-ip> netmask 255.255.255.0 pointopoint <destination-ip>

For example:

Machine 1

/sbin/ifconfig eth0 inet 192.168.0.10 netmask 255.255.255.0 pointopoint 192.168.0.11


Machine 2

/sbin/ifconfig eth0 inet 192.168.0.11 netmask 255.255.255.0 pointopoint 192.168.0.10

ping 192.168.0.10 # This should work now if your network cable is connected properly

Debug


/sbin/ethtool eth0 # Should show correct speed and link detected should be "yes"


Reference

http://docstore.mik.ua/orelly/networking/tcpip/ch06_01.htm

Cleaning Memory on Linux

If your system shows a lot of inactive memory or Swap space in use and you wish to cleanup before starting next job, then here is what you can do:

$ sync

# Cleanup memory held-up in Inactive and return to Unused, also removes unused pages/inodes/delentries from used memory
$ echo 3 > /proc/sys/vm/drop_caches

# Empty used up Swap space
# Disable swap and then re-enable
# This method is an aggressive one and if there isn't enough RAM free to hold up required pages from Swap, the system may crash
$ /sbin/swapoff -a
$ /sbin/swapon -a

Linux Tools

This list will continue to grow as I come across useful tools in Linux. I have also listed the command-line that I most frequently use:

ps - get the process information

• ps -flycae # will give information on scheduling class and kernel function name where process is sleeping

mcelog - Machine check log

• Determine system errors, decode, and log in syslog like /var/log/messages
  mcelog --dmi --syslog
  PS: The decoded hardware module is sometimes not helpful in determining the module at fault

OProfile - Profiling Linux system

• Covered under separate post here. 

mdadm - Manage Software  RAID

• Use mdadm for software mirroring devices:
  mdadm -C --level=raid1 --raid-devices=2 /dev/md0 /dev/sdb1 /dev/sdc1
  mdadm --manage /dev/md0 --fail /dev/sdc1
  mdadm --manage /dev/md0 --remove /dev/sdc1
  mdadm --manage --stop /dev/md0
  For striping, better to use LVM

LVM - Logical Volume Manager

• pvcreate <list of devices>
• vgcreate <vg-name> <list of pv devices>
• lvcreate -i<#of vg devices to stripe across> -I<stripe-size-default64k> -L<size-in-MB> -n<volume-name> <vg-name>

MUTT - The Mutt Mail User Agent

• echo "<message-body>" | mutt -a <attachment-file> -s "<subject>" <To email> [-b <BCC email>] [-c <CC email>] 

Linux Kernel Parameters

Excellent collection of available kernel boot parameters including sysctl options is available on Aarom Maxwell's page grouped by kernel versions starting 2.6.12.

kernel.org documentation is also an excellent resource for Linux kernel options.

Disable SELinux

Check current status:

cat /selinux/enforce

1 => enabled

0 => disable

To disable SELinux:

• Disable immediately withoput reboot, change will be lost upon reboot:
  echo 0 > /selinux/enforce
• Reboot required:
  • Set selinux=disabled in /etc/selinux/config, then reboot
  • In grub boot command-line add: selinux=0, and reboot

[Ref] Excellent post with details available here

Disable NIS in RHEL / Oracle Linux

Related configuration files:
/etc/host.conf - lists the lookup order. No change necessary.
/etc/nsswitch.conf - remove nis from hosts list
/etc/sysconfig/network - remove NISDOMAIN entry

Reboot.

Ethtool - Linux Networking tool

ethtool

See ethtool for the details. Here are a few frequently used ones:

# Get general info and connectivity status

/sbin/ethtool eth0

# Get driver info

/sbin/ethtool -i eth0  

# Get offload info

/sbin/ethtool -k eth0

# Set offload params. Eg. disable tcp and udp checksumming

/sbin/ethtool -K eth0 tx off

# To check checksum status:

tcpdump -vv -n -i eth0

Updating Time Zone in Linux

Following should work on RHEL, Oracle Linux, Centos, Ubuntu, and similar may work on others:
Update /etc/sysconfig/clock with the desired timezone from /usr/share/zoneinfo
Backup current localtime:
mv /etc/localtime /etc/localtime.o
cp /usr/share/zoneinfo/ /etc/localtime

If the command is available, the above can be achieved by using:
system-config-date

Configuring RSH without Password

Tried on RHEL and Oracle Linux:

[1] Install rsh-server*.rpm from the Linux distribution for your system or using yum.

[2] Add the following to /etc/securetty:
rsh
rexec
rlogin

[3] Edit the rsh, rexec, and rlogin files in /etc/xinetd.d/ and change value of disable from yes to no

[4] /etc/init/d/xinetd restart

[5] Add list of hosts to /etc/hosts as:


[6] Add the hosts to ~/.rhosts as:


[7] chmod 600 ~/.rhosts
Also, ensure .rhosts is owned by the right user:group. Otherwise use chown as well.

[8] Repeat steps 1 to 7 on all hosts

Now, you should be able to rsh among the hosts on which you have setup rsh-server correctly.

SSH without Password

In order to ssh from Host1 to Host2 without password, the following needs to be done:

user1@Host1:~> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa):
Created directory '/home/user1/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user1/.ssh/id_rsa.
Your public key has been saved in /home/user1/.ssh/id_rsa.pub.

If ~/.ssh doesn't exist on Host2 as user2, create one:

user2@Host2:~> mkdir -p .ssh
user2@Host2:~> chmod 700 .ssh

Append user1's public key to authorized_keys and authorized_keys2 in user2@Host2:.ssh/:

user1@Host1:~> cat .ssh/id_rsa.pub | ssh user2@Host2 'cat >> .ssh/authorized_keys'
user1@Host1:~> cat .ssh/id_rsa.pub | ssh user2@Host2 'cat >> .ssh/authorized_keys2'

user2@Host2:~> chmod 640 .ssh

Now ssh from user1@Host1 to user2@Host2 should be possible without password:
user1@Host1:~> ssh user2@Host2\

Avoid Host Verification

ssh -o "StrictHostKeyChecking no" user@host

Debugging

Set LogLevel to DEBUG in /etc/ssh/sshd_config
/etc/init.d/sshd restart
Then try to ssh, debug messages will be logged to /var/log/secure
PS: Once you have resolved the issue remember to switch back LogLevel to INFO and again restart sshd.

Editing Xen System.img

This post is useful when you want to change system configuration files of a stopped guest VM without botting up the guest first. This method can be used especially when your guest VM crashes or hangs during boot up due to a configuration error.


The following steps have been tested on Xen 4 and Oracle VM 2.2 as well for a Linux guest VM. From DOM0: 


Check FS Type
fdisk -u -l System.img

If FS type is ext3, you can directly mount a partition using lomount, otherwise follow the mounting LV's steps.

Mount Ext3 Partitions

Linux system.img with ext3 partitions can be mounted as:
lomount -diskimage System.img -partition 2 /mnt

Mounting LV's

# Mounting guest's root partition locally
# Find a free loop device
loop_dev=`losetup -f`

# Now bind the image file to that loop device
losetup ${loop_dev} System.img

# Next, scan the loop device for partitions
kpartx -av ${loop_dev}

# If /dev/mapper doesn't list LVs for the partitions from kpartx, find LV:
vgscan
vgchange -ay sysvg

# Mount the desired LVM
mount /dev/mapper/loop0p2 /mnt

LVs UnMounting
# After editing in /mnt, unmount and remove partitions:
umount /mnt

# Disable the LV
vgchange -an /dev/mapper/loop0p2

# Remove the discovered partitions
kpartx -dv ${loop_dev}

# Delete the loop device
losetup -d ${loop_dev}