FastCGI with Apache2.2

Install FastCGI (FCGI) on RHEL5 / Oracle Linux 5

1. Install httpd rpm along with dependencies. The following packages (and any dependencies) will have to be installed:

apr
postgresql-libs
apr-util

httpd

2. Install the following additional packages to build and install fcgi module:

expat-devel
cyrus-sasl-devel
openldap-devel

db4-devel

httpd-devel (this package will install the required /usr/lib64/httpd/build/special.mk and also installs a useful apxs utility)

3. Untar mod_fcgi 2.4.6 in /root

make top_dir=/usr/lib64/httpd

make top_dir=/usr/lib64/httpd install

Debugging
If you encounter:
Missing /usr/lib64/httpd/build/special.mk
Ensure that httpd-devel was installed:
rpm -qa|grep httpd-devel

Linux rpm commands

Following are the rpm commands I regularly use on RHEL and variants:

Command	Description	Example
rpm -qa	List all installed rpms	rpm -qa | grep kernel
rpm -qlm <rpm file> or rpm -qpl <rpm file>	List the files contained in the .rpm file
rpm -ivh <rpm file>	Check for dependency then install rpm
rpm -Uvh <rpm file>	Check for dependency then update package
rpm2cpio <rpm file> | cpio -idmv <leave empty for all or specify specific files in the rpm>	Extract files contained in the rpm (without install) into current directory

PostgreSQL Permission Denied

To debug and fix issues related to permission problem with PostgreSQL, especially so when creating tablespaces, the following is useful:

• Disable firewalls and selinux
• Check which user is postgres server running as. 
• Assuming it is running as postgres user, login as user postgres
• Check "id" to ensure user postgres is part of group postgres 
• Create the tablespace directory at the required location using mkdir. If you get permission error, fix that and retry. Do not create the directory as root.

The problem should be fixed.

Please leave a comment to share what worked for you to resolve PostgreSQL permission problems. Thanks!

Linux: Which process is listening on a port?

Under various circumstances, like when starting up a server or a service, you may encounter "port already in use" error. Here are different ways to determine which process is using that port.

• netstat -tulpn | grep <portNumber>
• fuser <port#>/tcp
• lsof -i tcp:<port#>

To determine if the commands are available on your system, try:
whereis <command>
e.g. whereis fuser

Once the pid is known for the process using the port, get information on the running process along with its command line:
ps -aux | grep <pid>

Reference

http://www.cyberciti.biz/faq/what-process-has-open-linux-port

OutOfMemoryError: Unable to create new native thread

This post suggests how to fix OutOfMemmory (OOM) error with Java on Linux. There are 2 things to try:

[1] Reduce the stack size in JVM command line and for Linux. RHEL/Oracle Linux default is 8096k to 10240k, while even for most enterprise applications as low as 128k may suffice. HotSpot JDK7 seems to suggest a minimum of 160k. So:

• Add "-Xss160k" to the JVM commandline
• Set "ulimit -s 160" from the Linux shell and confirm with "ulimit -a". This setting is per user session. To make it persist for a user, add it to /etc/security/limits.conf

[2] Track the total number of threads on the system:

watch 'ps -efL|wc -l'

Check the system limit:

cat /proc/sys/kernel/pid_max # Usually 32768

If the total number of threads reaches this pid_max, JVM will throw OOM with unable to create new native threads error. 

To fix this issue:

• Increase the system limit as root:
  echo 65536 > /proc/sys/kernel/pid_max 
• Or, to persist across reboots, add the following to /etc/sysctl.conf:
  kernel.pid_max = 65536

Vim - Matching Braces

Matching {}, (), [], /* */, can be very helpful when using vi to read a program code.

In command mode, take your cursor to one of the start/end braces or comment mark and press %, that is, Shift + 5 on most keyboards.

Another useful trick is to highlight the code between the matching tags, and can be achieved simply by:
:noremap % v%

References

http://vim.wikia.com/wiki/Moving_to_matching_braces

SVN Most Frequently Used Commands

Legend: bold => command; <> => optional option; [] => required option

Check the status of your local copy
svn status
Sync up with the repository
svn update
Check history of commit
svn log <dir/file>
Check history of commits with list of affected files
svn log -v <dir/file>
Diff local copy with latest repository version
svn diff <file/dir>
Diff between two revisions
svn diff -r [rev1]:[rev2] <file/dir>
Diff ignoring Ctrl+M characters at the end of line
svn diff -x --ignore-eol-style <file/dir>
Commit
svn commit -m "[comment]" <file/dir>
Add new files
svn add <file/dir>

Linux Point to Point Network Connection

Point to point connection is a network connection between 2 machines without a switch in between. Basically, both the ends of the cables are directly inserted into the 2 machines to be connected via p2p.

Advantages of p2p include avoiding routing overhead from the switch, simplicity, and to security.

Configuration is simply:

/sbin/ifconfig <dev> inet <local-host-ip> netmask 255.255.255.0 pointopoint <destination-ip>

For example:

Machine 1

/sbin/ifconfig eth0 inet 192.168.0.10 netmask 255.255.255.0 pointopoint 192.168.0.11


Machine 2

/sbin/ifconfig eth0 inet 192.168.0.11 netmask 255.255.255.0 pointopoint 192.168.0.10

ping 192.168.0.10 # This should work now if your network cable is connected properly

Debug


/sbin/ethtool eth0 # Should show correct speed and link detected should be "yes"


Reference

http://docstore.mik.ua/orelly/networking/tcpip/ch06_01.htm

Cleaning Memory on Linux

If your system shows a lot of inactive memory or Swap space in use and you wish to cleanup before starting next job, then here is what you can do:

$ sync

# Cleanup memory held-up in Inactive and return to Unused, also removes unused pages/inodes/delentries from used memory
$ echo 3 > /proc/sys/vm/drop_caches

# Empty used up Swap space
# Disable swap and then re-enable
# This method is an aggressive one and if there isn't enough RAM free to hold up required pages from Swap, the system may crash
$ /sbin/swapoff -a
$ /sbin/swapon -a

Linux Tools

This list will continue to grow as I come across useful tools in Linux. I have also listed the command-line that I most frequently use:

ps - get the process information

• ps -flycae # will give information on scheduling class and kernel function name where process is sleeping

mcelog - Machine check log

• Determine system errors, decode, and log in syslog like /var/log/messages
  mcelog --dmi --syslog
  PS: The decoded hardware module is sometimes not helpful in determining the module at fault

OProfile - Profiling Linux system

• Covered under separate post here. 

mdadm - Manage Software  RAID

• Use mdadm for software mirroring devices:
  mdadm -C --level=raid1 --raid-devices=2 /dev/md0 /dev/sdb1 /dev/sdc1
  mdadm --manage /dev/md0 --fail /dev/sdc1
  mdadm --manage /dev/md0 --remove /dev/sdc1
  mdadm --manage --stop /dev/md0
  For striping, better to use LVM

LVM - Logical Volume Manager

• pvcreate <list of devices>
• vgcreate <vg-name> <list of pv devices>
• lvcreate -i<#of vg devices to stripe across> -I<stripe-size-default64k> -L<size-in-MB> -n<volume-name> <vg-name>

MUTT - The Mutt Mail User Agent

• echo "<message-body>" | mutt -a <attachment-file> -s "<subject>" <To email> [-b <BCC email>] [-c <CC email>] 

OProfile: System / JIT Profiling on Linux

Requirements

• Oprofile tar
• kernel-debuginfo-`uname -r`.rpm
• kernel-debuginfo-commons-`uname -r`.rpm

Setup

Install kernel-debuginfo rpms for the kernel you are running.

As root user:

$ tar zvxf oprofile-0.9.7.tar
$ cd oprofile-0.9.7
$ CFLAGS="-m64" CXXFLAGS="-m64" ./configure \

   --with-java={my_jdk_installdir} \
   --libdir=/usr/local/lib64

Ensure that configure has passed, otherwise resolve any issues with missing headers and libraries and rerun configure as above. One common issue people run into on RHEL 5:
checking libiberty.h usability... no
checking libiberty.h presence... no
checking for libiberty.h... no
checking for cplus_demangle in -liberty... no Solution: Install binutils-devel rpm 

$ make
$ make install

Profiling JIT
For JIT profiling, the following option needs to be added to the Java command-line:

-agentpath:<libdir>/libjvmti_oprofile.so

OR

-agentlib:jvmti_oprofile

Also, add <oprof_install_dir>/lib/oprofile to LD_LIBRARY_PATH

[See Ref]   

Collection

$ opcontrol --reset
$ opcontrol --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux
$ opcontrol --session-dir=$DIR
$ opcontrol --start
$ opcontrol --stop
$ opcontrol --dump
$ opcontrol --save <session-name>
$ opcontrol --shutdown

Report

$ opreport session:<session-name>
$ opreport session:<session-name> --image-path /lib/modules/`uname -r`/kernel [--details | -d]
$ opreport session:<session-name> --image-path /lib/modules/`uname -r`/kernel [--symbols | -l]
$ opreport session:<session-name> [--callgraph | -c]

References

• oprofile.sourceforge.net [JVMTI]

Java EE 7: Moving into the Cloud

Updated: Java EE 7 launch webcasts:

For quick reference, this post lists the JSRs planned for the Java EE 7, slated to be released in Q2 of 2013:

• JSR 107 - JCACHE: Java Temporary Caching API 
• JSR 236 - Concurrency Utilities for Java EE 
• JSR 338 - Java Persistence 2.1
• JSR 339 - JAX-RS 2.0: The Java API for RESTful WebServices
• JSR 340 - Java Servlet 3.1 Specification
• JSR 341 - Expression Language 3.0
• JSR 342 - Java EE 7 Specification
• JSR 343 - JMS 2.0
• JSR 344 - JSF 2.2
• JSR 345 - EJB 3.2
• JSR 346 - Contexts and Dependency Injection for Java EE 1.1
• JSR 349 - Bean Validation 1.1
• JSR 350 - Java State Management
• JSR 352 - Batch Applications for the Java Platform
• JSR 353 - Java API for JSON Processing

Post on Java EE 6 is here.


Reference:
1. Jdevelopment 
2. Java Magazine

	Look inside >
Looking Ahead to Java EE 7

3. Java EE 7 |

Java EE 6

• JSR 196 - Java Authentication SPI for Containers
• JSR 236 - Concurrency Utilities (including Timer)
• JSR 237 - Work Manager for Application Servers
• JSR 299 - Web Beans 1.0 
• JSR 311 - JAX-RS: Java API for RESTful Web Services
• JSR 314 - Java Server Faces 2.0  
• JSR 315 - Servlet 3.0
• JSR 316 - Java EE 6 Specification
• JSR 317 - Java Persistence 2.0
• EJB 3.1

Post on Java EE 7 is here.

References:

• Oracle Java EE 6 | New Features in Java EE 6 | 
• Hands on Lab 
• Using CDI with JSF | CDI Tutorial
• CDI Bootstrapping | Adding |  Tutorial Part I |  II | III
• Dependency Injection 
• SOAP vs. REST 

GlassFish Port Permission error

Symptoms

When trying to create a domain, permission warning may appear as follows:

/opt/glassfish3/glassfish/bin/asadmin --user admin --passwordfile /opt/glassfish3/passwordfile create-domain --adminport 4848 --instanceport 8000 myDom1

You do not have permission to use port 4848 for myDom1. Try a different port number or login to a more privileged account.CLI130 Could not create domain, myDom1Command create-domain failed.

Similarly, when starting a domain, the following may be encountered:

There is a process already using the admin port 4848 -- it probably is another instance of a GlassFish server.Command start-domain failed.

Possible solutions

• Ensure port is not in use, check "netstat -an | grep 4848" 
• "ps -ef|grep java" to check if another GF instance is running. Shut it down first.
• Identify 'hostname' 
• Verify that /etc/sysconfig/network has the correct hostname
• Ensure 'hostname' is in /etc/hosts and assigned to the correct IP address, where the server will listen

Linux Kernel Parameters

Excellent collection of available kernel boot parameters including sysctl options is available on Aarom Maxwell's page grouped by kernel versions starting 2.6.12.

kernel.org documentation is also an excellent resource for Linux kernel options.