Tuning TCP/IP stack in Linux

Kernel tunable parameters for TCP/IP stack performance

Tunable parameter	Default value	Option description
/proc/sys/net/core/rmem_default	"110592"	Defines the default receive window size; for a large BDP, the size should be larger.
/proc/sys/net/core/rmem_max	"110592"	Defines the maximum receive window size; for a large BDP, the size should be larger.
/proc/sys/net/core/wmem_default	"110592"	Defines the default send window size; for a large BDP, the size should be larger.
/proc/sys/net/core/wmem_max	"110592"	Defines the maximum send window size; for a large BDP, the size should be larger.
/proc/sys/net/ipv4/tcp_window_scaling	"1"	Enables window scaling as defined by RFC 1323; must be enabled to support windows larger than 64KB.
/proc/sys/net/ipv4/tcp_sack	"1"	Enables selective acknowledgment, which improves performance by selectively acknowledging packets received out of order (causing the sender to retransmit only the missing segments); should be enabled (for wide area network communication), but it can increase CPU utilization.
/proc/sys/net/ipv4/tcp_fack	"1"	Enables Forward Acknowledgment, which operates with Selective Acknowledgment (SACK) to reduce congestion; should be enabled.
/proc/sys/net/ipv4/tcp_timestamps	"1"	Enables calculation of RTT in a more accurate way (see RFC 1323) than the retransmission timeout; should be enabled for performance.
/proc/sys/net/ipv4/tcp_mem	"24576 32768 49152"	Determines how the TCP stack should behave for memory usage; each count is in memory pages (typically 4KB). The first value is the low threshold for memory usage. The second value is the threshold for a memory pressure mode to begin to apply pressure to buffer usage. The third value is the maximum threshold. At this level, packets can be dropped to reduce memory usage. Increase the count for large BDP (but remember, it's memory pages, not bytes).
/proc/sys/net/ipv4/tcp_wmem	"4096 16384 131072"	Defines per-socket memory usage for auto-tuning. The first value is the minimum number of bytes allocated for the socket's send buffer. The second value is the default (overridden by wmem_default) to which the buffer can grow under non-heavy system loads. The third value is the maximum send buffer space (overridden by wmem_max).
/proc/sys/net/ipv4/tcp_rmem	"4096 87380 174760"	Same as tcp_wmem except that it refers to receive buffers for auto-tuning.
/proc/sys/net/ipv4/tcp_low_latency	"0"	Allows the TCP/IP stack to give deference to low latency over higher throughput; should be disabled.
/proc/sys/net/ipv4/tcp_westwood	"0"	Enables a sender-side congestion control algorithm that maintains estimates of throughput and tries to optimize the overall utilization of bandwidth; should be enabled for WAN communication. This option is also useful for wireless interfaces, as packet loss may not be caused by congestion.
/proc/sys/net/ipv4/tcp_bic	"1"	Enables Binary Increase Congestion for fast long-distance networks; permits better utilization of links operating at gigabit speeds; should be enabled for WAN communication.

References

• http://www.ibm.com/developerworks/library/l-hisock/
• http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf
• http://www.ntop.org/products/pf_ring/hardware-packet-filtering

Show DDL for create Oracle DB table

Default long size is only 80. If your DDL is longer you will have to increase the long size to display full DDL. The GET_DDL may not exactly match what was used for creating the table as it will list the default optional settings as well that the actual create table script may have omitted. 

SQL> set long 200000

SQL> select DBMS_METADATA.GET_DDL('TABLE','<your_table_name>') from dual;

Database benchmarks

Standard OLTP Benchmarks

• TPC-C/E
• SPECjEnterprise2010

Open source / Free OLTP Benchmarks

• HammerDB - Workload provides TPC-C/H like transactions, not to be directl compared with TPC-C/H. Use New Orders Per Min (NOPM) metric to compare different DB vendors.. It supports Oracle, DB2, SQL server, MySQL/MariaDB, Aurora, Redshift, etc. Requires GUI.
• YCSB - Workloads are mostly SELECT/UPD/INS combinations for variious data stores:
  A—Update heavy Read: 50% Update: 50% - Zipfian - Session store recording recent actions in a user session
  B—Read heavy Read: 95% Update: 5% - Zipfian - Photo tagging; add a tag is an update, but most operations are to read tags
  C—Read only Read: 100% - Zipfian - User profile cache, where profiles are constructed elsewhere (e.g., Hadoop)
  D—Read latest Read: 95% Insert: 5% -  Latest - User status updates; people want to read the latest statuses 
  E—Short ranges Scan: 95% Insert: 5% - Zipfian/Uniform (Workload E uses the Zipfian distribution to choose the first key in the range, and the Uniform distribution to choose the number of records to scan) - Threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id)
  
  JDBC version is available that can be used for RDBMS like Oracle, MySQL, etc.
  
• Sysbench - Includes oltp (read/writes), fileio, cpu, memory, threads, mutex workloads. 
• Swingbench - Includes 6 workloads - OrderEntry (JDBC or PL/SQL variants), SalesHistory, TPC-DS Like, JSON, CallingCircle and StressTest
• Orion Calibration - Tool for Oracle DB, simulating I/O workloads using same I/O software stack as Oracle. 
• iobench - Java benchmark targeting EC2 and EBS

Disable Password Expiry on Linux

chage -I -1 -m 0 -M 99999 -E -1

Port forwarding in Linux

To access a machine not directly accessible on network, port forwarding can be used from an accessible machine to the destination machine. 

One common scenario where port forwarding is useful is where a process, like web server, is on a machine with no x-window/browser and it is not directly accessible from the machine running the browser, but an accessible "gateway" machine is available in between.

The following can be run on the "gateway" machine:

ssh <user>@<gateway-host> -L <port>:<destination-host>:<destination-port> -g -N

As long as this process is running, gateway-host:port will show the content from <destination-host>:<destination-port>. 

Oracle Certified Professional Java SE 8 Upgrade Exam

The certification for Java SE 8 is not yet available, but here are the "What's New" from the JDK 8 release, and are candidates for the upgrade exam preparation:

• Lambda expressions
• Default methods in interface
• Aggregate operations
• Date-Time
• Type annotations
• Repeating annotations
• New java.util.stream package

There are several other updates in Security, JavaFX, Tools, etc., but we will wait and see if they make it to the upgrade certification syllabus.

Details on Oracle certification can be accessed from the Certification tab on http://education.oracle.com

 

References:

• What's New in JDK 8 
• Java Tutorial

Creating SVN for Existing or New Project

Step 1: Create a directory where the repository will be maintained
mkdir /opt/svnrepos
cd /opt/svnrepos
svnadmin create projectName

Step 2: Add project to SVN
#Create project directory if it does not exist
mkdir /home/user/projectName
svn import /home/user/projectName file:///opt/svnrepos/projectName -m "Initial commit of project"

Step 3: Checkout project and start using SVN
cd /home/user/workspace
svn co file:///opt/svnrepos/projectName
cd projectName
svn status
svn log
For more frequently used SVN commands, see previous post here.

Difference in interrupt counts from /proc/interrupts

The following python script can be used to determine interrupt counts occurred between the before and after "cat /proc/interrupts":

#!/usr/bin/python
'Interrupt counts between before and after /proc/interrupts snapshots'

import sys
from itertools import izip

def checkUsage():
    #Check Usage
    if len(sys.argv) < 3:
        print "Usage ->"
        print sys.argv[0]," </proc/interrupts before> </proc/interrupts after>"

def readFiles(fileName1, fileName2):
    fileA = open(fileName1)
    fileB = open(fileName2)
    for lineA, lineB in izip(fileA, fileB):
        wordsA = lineA.rstrip().split()
        wordsB = lineB.rstrip().split()
        for wordA, wordB in izip(wordsA, wordsB):
            if wordA.isdigit() and wordB.isdigit():
                sys.stdout.write("%s%s" % (str(int(wordB) - int(wordA)),"\t"))
            else:
                sys.stdout.write("%s%s" % (wordB," "))
        print

def main():
    checkUsage()
    #readFile(sys.argv[1])
    readFiles(sys.argv[1], sys.argv[2])

#Don't execute if script is imported instead of executed
if __name__ == '__main__':
    main()

Please leave me a comment if you find this post helpful.

Oracle RDBMS tips

Monitoring

Tables/Indexes

• desc user_tables
• desc user_indexes

DB CPU

• V$SYSSTAT: Oracle CPU usage for all sessions. The parse time cpu statistic shows the total CPU time used for parsing.
• V$SESSTAT: Oracle CPU usage for each session. 
• select * from v$resource_limit;
• select count(*) from V$process;

Wait Events

• db file sequential read: usually single block INDEXED reads [1]. Faster disks or more spindles will help. Larger buffer cache may help.
• db file scattered read
• db file single write
• db file parallel write
• log file parallel write

Latches

• HW enq
• Latch: Cache Buffer Chain - Ref [2]
  -- Find memory address:
  

  SELECT 
        addr,
        sleeps
  FROM 
        v$latch_children c,
        v$latchname n
  WHERE
        n.name='cache buffers chains' and
        c.latch#=n.latch# and
        sleeps > 100
  ORDER BY sleeps;
  
  -- Find file and block numbers:
  SELECT file#, dbablk, class, state, TCH
    FROM X$BH
    WHERE HLADDR='address of latch';
  -- Find segment name from dba_segments
  

Data/Log/Sys Files

• select name from v$datafile;
• select name from v$controlfile;
• select member from v$logfile;

Tablespace Sizes

select TABLESPACE_NAME, sum(decode(AUTOEXTENSIBLE, 'YES', MAXBYTES, BYTES))/1024/1024/1024 MAX_SIZE_IN_GB, count(*) NUM_TABLES

from DBA_DATA_FILES

group by TABLESPACE_NAME;

Performance Tuning Recommendations

• #Transactions = based on the application, the number of concurrent transactions that can occur
• #Sessions = (1.1 * PROCESSES) + 5
• DML_LOCKS = #Transactions * (avg #SQLs per transaction)
• REPLICATION_DEPENDENCY_TRACKING: disable where replication is not required as skipping tracking generation will make read/write operations faster  

References

[1] http://asktom.oracle.com/pls/asktom/f?p=100:11:0:::%3AP11_QUESTION_ID:6265095774206
[2] https://docs.oracle.com/cd/B16240_01/doc/doc.102/e16282/oracle_database_help/oracle_database_wait_bottlenecks_latch_cache_buffer_chains_pct.html

Avoiding SSH delay

# Enable verbose mode in ssh to determine where the hang is

[local-host]$ ssh -v <remote-host>

It will result in the following:

debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to <remote-host> [<remote-IP>] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: loaded 2 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client <port1> <port2> none
debug1: kex: client->server <port1> <port2> none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '<remote-host>' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:33
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Next authentication method: password
root@<remote-host>'s password:

Sometimes the ssh appears to hang / wait for a while at the line in bold above. It is usually due to the UseDNS - default is yes. Disabling it and restarting sshd on the remote-host should fix this delay in ssh:

Add the following line in bold to /etc/ssh/sshd_config on remote-host:

#      UseDNS  Specifies whether sshd should look up the remote host name and
#      check that the resolved host name for the remote IP address maps
#      back to the very same IP address.The default is "yes".
UseDNS no

Restart sshd on the remote-host:

[remote-host]$ service sshd restart

GIT commonly used commands

GIT Structure

• Central repo (remote-repo)
• Local repo (local-repo): Local copy of remote-repo. If there are new pushes to remote-repo, local-repo will be behind. If you have unpushed checkins, local-repo will be ahead of remote-repo.
• Local index (index): Changed or new Files need to be staged/added to index, before it can be checked in.
• Personal workspace (ws): Working copy of the local-repo.  

Creating local-repo and ws for the first time

• git branch -a # Show all branches including remote
• git checkout <branch> # Switch to branch
• git status # Check current branch
• git log # commit/push history
• git pull origin <branch> # pull branch from remote to current branch. To update current ws branch, use same branch name.

Sequence of Steps

• pull / rebase [remote to ws]
• git pull origin <branch> # pull branch from remote to current branch. To update current ws branch, use same branch name.
• fetch [remote to local]
• checkout HEAD [local to ws]
• checkout [index to ws]
• diff HEAD [b/w ws and local]
• diff [b/w ws and index]
• add (-u) [ws to index]
• commit [index to local]
• commit -a [ws to local]
• push [local to repo]
• git push origin <branch> # To push current ws branch, use same branch name. Default is all local branches.

Review changes

• Commit history: git log
• Last commit changes: git show
• Commit history with changes: git log -p 
• Check if a commit is merged to current branch: git branch --contains  <6 digits of commit number>

Revert local changes

• git checkout <specific-file>
• git checkout -- . # all unstaged changes will be lost 
• git reset HEAD <file> or git rm --cached # to remove file from commit. reset is the preferred way as rm without --cached could delete the file loosing changes


Merge

• git checkout main-branch
• git pull to latest...
  
• git merge side-branch
• resolve conflicts manually, see git status and do git add <resolved file>
• git branch -d <local branch>
• git push origin :remote-branch     # : implies delete remote branch

Stash

• git stash
• git stash list
• git stash pop
• git stash drop

Alias
To use git <alias_name>:

• git config --global alias.unstage 'reset HEAD --'; 
• git unstage myfile
• git config --global alias.uncommit 'reset --soft HEAD^';

Delete/Cleanup Remote/Local branch

• git remote prune origin // remove invalid local references to deleted remote branch
• git gc --prune=now // local cleanup
• git branch -d trac29-consol-wsdl / /delete local branch

Self Signed SSL Certificate

Generate RSA Private Key
openssl genrsa -des3 -out server.key 1024

Generate Certificate Signing Request (CSR)
openssl req -new -key server.key -out server.csr


Remove Passphrase from Key
cp server.key server.key.org
openssl rsa -in server.key.org -out server.key


Generate Self-Signed Certificate
openssl x509 -req -days 5000 -in server.csr -signkey server.key -out server.crt


Verify/Install mod_ssl
Requires mod_ssl be installed for Apache

Look for mod_ssl.so in httpd/modules dir or rpm -qa |grep mod_ssl

Otherwise, use yum install mod_ssl or download mod_ssl rpm and install.

mod_ssl does have dependency on distcache, so distcache also need to be installed

Install Private Key and Certificate
<httpd-conf> could be different based on Apache installation. For example, /etc/httpd/conf
cp server.crt <httpd-conf>/ssl.crt
cp server.key <httpd-conf>/ssl.key


Configure SSL Enabled Virtual Hosts
When mod_ssl is installed it creates a ssl.conf, usually in /etc.httpd/conf.d. Verify that you have the following entries:

SSLEngine on
SSLCertificateFile <httpd-conf>/ssl.crt/server.crt
SSLCertificateKeyFile <httpd-conf>/ssl.key/server.key
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
CustomLog logs/ssl request log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"


Restart Apache and Test

https://<Server_Host>:<Port>

Reference

http://www.akadia.com/services/ssh_test_certificate.html

FastCGI with Apache2.2

Install FastCGI (FCGI) on RHEL5 / Oracle Linux 5

1. Install httpd rpm along with dependencies. The following packages (and any dependencies) will have to be installed:

apr
postgresql-libs
apr-util

httpd

2. Install the following additional packages to build and install fcgi module:

expat-devel
cyrus-sasl-devel
openldap-devel

db4-devel

httpd-devel (this package will install the required /usr/lib64/httpd/build/special.mk and also installs a useful apxs utility)

3. Untar mod_fcgi 2.4.6 in /root

make top_dir=/usr/lib64/httpd

make top_dir=/usr/lib64/httpd install

Debugging
If you encounter:
Missing /usr/lib64/httpd/build/special.mk
Ensure that httpd-devel was installed:
rpm -qa|grep httpd-devel

Linux rpm commands

Following are the rpm commands I regularly use on RHEL and variants:

Command	Description	Example
rpm -qa	List all installed rpms	rpm -qa | grep kernel
rpm -qlm <rpm file> or rpm -qpl <rpm file>	List the files contained in the .rpm file
rpm -ivh <rpm file>	Check for dependency then install rpm
rpm -Uvh <rpm file>	Check for dependency then update package
rpm2cpio <rpm file> | cpio -idmv <leave empty for all or specify specific files in the rpm>	Extract files contained in the rpm (without install) into current directory

PostgreSQL Permission Denied

To debug and fix issues related to permission problem with PostgreSQL, especially so when creating tablespaces, the following is useful:

• Disable firewalls and selinux
• Check which user is postgres server running as. 
• Assuming it is running as postgres user, login as user postgres
• Check "id" to ensure user postgres is part of group postgres 
• Create the tablespace directory at the required location using mkdir. If you get permission error, fix that and retry. Do not create the directory as root.

The problem should be fixed.

Please leave a comment to share what worked for you to resolve PostgreSQL permission problems. Thanks!

Linux: Which process is listening on a port?

Under various circumstances, like when starting up a server or a service, you may encounter "port already in use" error. Here are different ways to determine which process is using that port.

• netstat -tulpn | grep <portNumber>
• fuser <port#>/tcp
• lsof -i tcp:<port#>

To determine if the commands are available on your system, try:
whereis <command>
e.g. whereis fuser

Once the pid is known for the process using the port, get information on the running process along with its command line:
ps -aux | grep <pid>

Reference

http://www.cyberciti.biz/faq/what-process-has-open-linux-port

OutOfMemoryError: Unable to create new native thread

This post suggests how to fix OutOfMemmory (OOM) error with Java on Linux. There are 2 things to try:

[1] Reduce the stack size in JVM command line and for Linux. RHEL/Oracle Linux default is 8096k to 10240k, while even for most enterprise applications as low as 128k may suffice. HotSpot JDK7 seems to suggest a minimum of 160k. So:

• Add "-Xss160k" to the JVM commandline
• Set "ulimit -s 160" from the Linux shell and confirm with "ulimit -a". This setting is per user session. To make it persist for a user, add it to /etc/security/limits.conf

[2] Track the total number of threads on the system:

watch 'ps -efL|wc -l'

Check the system limit:

cat /proc/sys/kernel/pid_max # Usually 32768

If the total number of threads reaches this pid_max, JVM will throw OOM with unable to create new native threads error. 

To fix this issue:

• Increase the system limit as root:
  echo 65536 > /proc/sys/kernel/pid_max 
• Or, to persist across reboots, add the following to /etc/sysctl.conf:
  kernel.pid_max = 65536

Vim - Matching Braces

Matching {}, (), [], /* */, can be very helpful when using vi to read a program code.

In command mode, take your cursor to one of the start/end braces or comment mark and press %, that is, Shift + 5 on most keyboards.

Another useful trick is to highlight the code between the matching tags, and can be achieved simply by:
:noremap % v%

References

http://vim.wikia.com/wiki/Moving_to_matching_braces

SVN Most Frequently Used Commands

Legend: bold => command; <> => optional option; [] => required option

Check the status of your local copy
svn status
Sync up with the repository
svn update
Check history of commit
svn log <dir/file>
Check history of commits with list of affected files
svn log -v <dir/file>
Diff local copy with latest repository version
svn diff <file/dir>
Diff between two revisions
svn diff -r [rev1]:[rev2] <file/dir>
Diff ignoring Ctrl+M characters at the end of line
svn diff -x --ignore-eol-style <file/dir>
Commit
svn commit -m "[comment]" <file/dir>
Add new files
svn add <file/dir>

Linux Point to Point Network Connection

Point to point connection is a network connection between 2 machines without a switch in between. Basically, both the ends of the cables are directly inserted into the 2 machines to be connected via p2p.

Advantages of p2p include avoiding routing overhead from the switch, simplicity, and to security.

Configuration is simply:

/sbin/ifconfig <dev> inet <local-host-ip> netmask 255.255.255.0 pointopoint <destination-ip>

For example:

Machine 1

/sbin/ifconfig eth0 inet 192.168.0.10 netmask 255.255.255.0 pointopoint 192.168.0.11


Machine 2

/sbin/ifconfig eth0 inet 192.168.0.11 netmask 255.255.255.0 pointopoint 192.168.0.10

ping 192.168.0.10 # This should work now if your network cable is connected properly

Debug


/sbin/ethtool eth0 # Should show correct speed and link detected should be "yes"


Reference

http://docstore.mik.ua/orelly/networking/tcpip/ch06_01.htm