Resolve grails jetty hangs problem caused by commons dbcp

Recently, I'm developing a website by grails.
Few days ago, I found a problem, the web server hangs after requesting the homepage.
The latest change is that the homepage is requesting 15+ articles instead of just 2 articles.
For development environment, the articles are stored in mysql.
The article is retrieved by requesting another link article controller,
and the related code in ArticleController.groovy looks like this,
... ...
def article = Article.get(id) //#1
if(article){
def info = articleService.retrieveInfo(article) //#2
}
... ...
code #1, finds the article instance from mysql db.
code #2, articleService is a grails service in grails-app/services directory.
the retrieveInfo method retrieves some info from the given article instance without any db operation.
The homepage should show 15+ articles, so the code above runs 15+ times in 15+ separate http request threads.

And then the grails embedded web server jetty HANGS. no more response.
After deploy the application in tomcat, tomcat hangs too.

What's the problem?
I guess it may due to thread deadlock, but how?
First, I got the java thread dump,

//redirect console output to a file
yichao src $ grails run-app > console.txt 2>&1

//after requesting homepage, press ctrl+\ (or kill -QUIT <pid>) to generate Java thread dump.
I found many threads are waiting on and locked the same monitor 0x96c6de90.
this is the code snippet, it's btpool0-15 thread, btpool0-14~2 threads are just the same.

"btpool0-15" prio=10 tid=0x08a76800 nid=0x22e4 in Object.wait() [0x8f2f2000..0x8f2f60c0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x96c6de90> (a org.apache.commons.pool.impl.GenericObjectPool)
at java.lang.Object.wait(Object.java:485)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:748)
- locked <0x96c6de90> (a org.apache.commons.pool.impl.GenericObjectPool)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:82)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
......
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:170)
at ArticleController.xxxx(ArticleController.groovy:85)
......
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:295)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:503)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:827)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:511)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:210)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:379)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:361)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

the ArticleController.groovy:85 code is the code #2 above.
This time, I found that in grails, when we define "boolean transactional = true" in ArticleService.groovy.
articleService.retrieveInfo() method opens a connection even if we have no db operation.

so every single thread tries to open mysql connection twice by code #1 and #2 above.
But we are just open connection and only select data from mysql, why hangs?
The code waits at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:748)

Thanks to apache, I can easily download the source code, and the line 748 is,
public Object borrowObject() throws Exception {
........
synchronized(this) {
........
switch(_whenExhaustedAction) {
........
case WHEN_EXHAUSTED_BLOCK:
try {
if(_maxWait <= 0) {
wait(); //#line 748
} else {
wait(_maxWait);
}
...........

see #line 748, we got this, in GenericObjectPool.borrowObject() method, if connections in pool exhausted, it will block.
The default active connections limit is 10, but we have 15+ thread, so all block!

Actually it should not to be like this. And Indeed it's already filed bug for apache commons dbcp 1.2.1.
The grails 1.0.3 is just using commons-dbcp-1.2.1.jar.
you can find this bug at https://issues.apache.org/jira/browse/DBCP-260

PS:

In grails, calling any methods in transactional-enabled services will try to open a db connection.
So only call methods in transactional-enabled services if they are related to db operation and really need transaction.
Put all other methods in transactional-disabled services or normal groovy in src/groovy/*.groovy files.
This resolves the problem above successfully.

BTW, The transaction is enabled by default in grails 1.0.3.

ZInfo - Information sharing platfrom on Android

ZInfo is finally released for Android Developer Challenge after our many hard working days.
It's an information sharing platform based on your contacts and location.
The video demo can be found at http://infosword.com/zinfo/video.html.

We gave a presentation about ZInfo on Beijing Open Party - Before the Rain.
It makes me the most excited that I meet a lot of open source followers.
They are very friendly, full of passion, enjoy coding! I learned a lot from them.

Special thanks to cleverpig!
He is so kind and want to sponsor us a server to host ZInfo.
And he understands us fully when we decided to put off releasing publicly recently.

It's so great to meet you guys in the community!

python script to email me the dynamic ip from tp-link router

At home, my boxes are connected to internet by Beijing ADSL.
when connected, Beijing ADSL gives me a dynamic ip and it changes about every 24 hours.
Before I get the dynamic ip by a service provided by oray.net.
However recently the service is broken.
So I have to write a python script to check the dynamic ip from my tp-link router every 3 minutes and if it changed, emails the new dynamic ip to my gmail.

I also write a bash script to run the above python script at boot on my gentoo linux.

The python script:

#!/usr/bin/python

import urllib
import urllib2
import libgmail
import cookielib
import sys
import re
import base64
import socket
import time
from urlparse import urlparse

nowip=''
logfile=open('/var/log/ipsender.log','w')

def log(s):
  now=time.strftime('%Y-%m-%d %X')
  logfile.write('%s %sn'%(now,s))
  logfile.flush()

def gmailsender(to_addr,subject,msg):
  username='xxx@gmail.com'
  password='xxx'
  ga=libgmail.GmailAccount(username,password)
  ga.login()
  gmsg=libgmail.GmailComposedMessage(to_addr,subject,msg)
  ga.sendMessage(gmsg)

def check():
  global nowip
  timeout=10
  socket.setdefaulttimeout(timeout)
  username = 'admin'
  password = 'xxx'
  try:
    theurl='http://192.168.1.1/userRpm/StatusRpm.htm'
    req = urllib2.Request(theurl)
    base64string = base64.encodestring('%s:%s' % (username, password))[:-1]
    authheader =  "Basic %s" % base64string
    req.add_header("Authorization", authheader)
    handle = urllib2.urlopen(req)
    str=handle.read()
    ss='var wanPara = new Array'
    start=str.index(ss)+ss.__len__()
    end=str.index(';',start)
    s2=str[start:end]
    log(s2)
    t=eval(s2)
    log(t[2])
    handle.close()
    if nowip!=t[2]:
      nowip=t[2]
      gmailsender('mygmail@gmail.com','ddnsip===%s'%nowip,s2)
      log('send %s successfully.'%nowip)
    else:
      log('ddnsip remains %s.'%nowip)
  except:
    log('Unexpected error:', sys.exc_info()[0])

def main():
  while True:
    check()
    time.sleep(3*60)

main()

The bash script put in /etc/init.d/:

#!/sbin/runscript
# Copyright 1999-2004 Gentoo Foundation
start()	{
	ebegin "Starting ipsender"
	nohup python /home/zeaster/amp/ipsender.py >> /var/log/ipsender-nohup.log&
	eend $?
}
stop()	{
	ebegin "Stopping ipsender unsuccessfully"
	eend $?
}

wake-on-lan by python script

How to wake up your computer on the LAN?
the answer is just to send a Magic Packet to your computer on port 7 or 9.
However when your computer is down, it has no ip, so send it to the broadcast ip
the following python script implements the above idea.

import struct, socket

def wake_on_lan(ether_addr, inet_addr):
  addr_byte=ether_addr.split(':')
  hw_addr=struct.pack('BBBBBB',
    int(addr_byte[0],16),
    int(addr_byte[1],16),
    int(addr_byte[2],16),
    int(addr_byte[3],16),
    int(addr_byte[4],16),
    int(addr_byte[5],16))
  msg='\xff'*6+hw_addr*16
 
  s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
  s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
  s.sendto(msg, (inet_addr,7))
  s.close()

wake_on_lan('00:E0:4C:E1:D9:CC', '192.168.1.255')

The code above is based on socket.SOCK_DGRAM to send UDP packet.
How to create a UDP packet based on raw socket in python?
the project dpkt does an excellent job.
the following example shows that:

import socket
from dpkt import ethernet,udp,ip
import dpkt
import struct
import string

iface = "eth1"
mac   = "00:0E:35:AB:9B:40"
inet  = "192.168.1.6"
debug = 0

def eth_aton(buffer):
    addr =''
    temp = string.split(buffer,':')
    buffer = string.join(temp,'')
    for i in range(0, len(buffer), 2):
        addr = ''.join([addr,struct.pack('B', int(buffer[i: i + 2], 16))],)
    return addr

def buildUdp():
    ether_addr='00:E0:4C:E1:D9:CD'
    addr_byte=ether_addr.split(':')
    hw_addr=struct.pack('BBBBBB',
       int(addr_byte[0],16),
       int(addr_byte[1],16),
       int(addr_byte[2],16),
       int(addr_byte[3],16),
       int(addr_byte[4],16),
       int(addr_byte[5],16))
    msg='xff'*6+hw_addr*16

    udp_p = udp.UDP()
    udp_p.sport = 0x8130
    udp_p.dport = 7
    udp_p.data = msg
    udp_p.ulen += len(udp_p.data)

    ip_p = ip.IP()
    ip_p.src = socket.inet_aton('192.168.1.6');
    ip_p.dst = socket.inet_aton('192.168.1.255');
    ip_p.off = 0x4000
    ip_p.data = udp_p
    ip_p.p = ip.IP_PROTO_UDP
    ip_p.len += len(udp_p)

    packet = ethernet.Ethernet()
    packet.src = eth_aton(mac)
    packet.dst = eth_aton('ff:ff:ff:ff:ff:ff')
    packet.data = ip_p
    packet.type = ethernet.ETH_TYPE_IP

    if debug: print dpkt.hexdump(str(packet))
    return packet

s = socket.socket(socket.PF_PACKET, socket.SOCK_RAW)
s.bind((iface,ethernet.ETH_TYPE_IP))
packet = buildUdp()
s.send(str(packet))

统一Mac OS X，Windows XP/Vista以及linux的系统时间

Windows把BIOS时间作为系统时间，而Mac OS X把BIOS时间作为GMT+0时间，所以对于生活在GMT+8时区的中国用户来说，这两个系统共存时系统时间是不一致的。
即：如果windows时间为12:00，则到Mac OS X下就变成了20:00，在Mac OS X下改过来后，再回到windows下就又错了。

之前看过这篇blog，讲了通过修改windows注册表，使得windows也把bios时间作为GMT+0时间，这样就可以解决这个问题了，具体操作如下：
在HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation\中加一项DWORD，RealTimeIsUniversal，把值设为1即可。

这个办法是靠调整windows时间设置来解决问题的，但是当系统中有linux时，因为linux也是使用bios时间作为系统时间，所以这时linux系统的时间就又不一致了。
最好的办法就是调整Mac OS X的时间。于是想到了强大的Bash Shell，先写脚本1用来开机时自动把时间校对准，然后关机时再运行脚本2，把时间调整回去。
脚本1：
#!/bin/bash
d=$(date "+%d")
H=$(date "+%H")
M=$(date "+%M")
S=$(date "+%S")
let "H=$H-16"

z=0
if [ "$H" -lt "$z" ]
then
  let "H=$H+24"
  let "d=$d-1"
fi

z=10

if [ "$d" -lt "$z" ]
then
  d=0$d
fi

if [ "$H" -lt "$z" ]
then
  H=0$H
fi

sudo date -u $d$H$M.$S

脚本2
#!/bin/bash
ss=$(date "+%d%H%M.%S")
sudo date -u $ss

那么如何实现自动开机运行以及关机自动运行呢？
1 开机自动运行
写shell，然后在applescript中调用shell，再把applescript保存为app，添加到login items中
注：我的.sh文件和textmate做绑定了，这样直接添加.sh文件不会执行shell，而是被textmate打开

2 关机自动运行
使用launchbar的restart和shutdown applescript，因此只需在这两个script中添加调用shell的一句即可了。

bash shell修改系统时间需要root权限，那么如何在applescript中调用时输入root密码呢？
使用如下命令：
do shell script "sudo date -u 1200" password "mypwd" with administrator privileges

xmlrpc for python

使用python处理xmlrpc太简单了，又一次感受到了python的力量！
下面以使用python调用wordpress提供的xmlrpc方法为例简单介绍一下：

1） how to call xmlrpc method in python
>>> import xmlrpclib
>>> from pprint import pprint
>>> server = xmlrpclib.ServerProxy("http://localhost/wordpress/xmlrpc.php")
>>> pprint(server.system.listMethods() )

['system.multicall',
 'system.listMethods',
 'system.getCapabilities',
 'demo.addTwoNumbers',
 'demo.sayHello',
 'pingback.extensions.getPingbacks',
 'pingback.ping',
 'mt.publishPost'......]

>>> blogs = server.metaWeblog.getRecentPosts('','admin','passwd',5)
>>> pprint(blogs)
>>> print(blogs[2]['permaLink'])

http://localhost/wordpress/?p=135

2）how to setup a xmlrpc server in python

import calendar, SimpleXMLRPCServer
#The server object
class Calendar:
    def getMonth(self, year, month):
        return calendar.month(year, month)

    def getYear(self, year):
        return calendar.calendar(year)
calendar_object = Calendar()
server = SimpleXMLRPCServer.SimpleXMLRPCServer(("localhost", 8888))
server.register_instance(calendar_object)
#Go into the main listener loop
print "Listening on port 8888"
server.serve_forever()

3）write a client to test server above
import xmlrpclib
server = xmlrpclib.ServerProxy("http://localhost:8888")
month = server.getMonth( 2002, 8 )
print month

     August 2002
Mo Tu We Th Fr Sa Su
                 1    2    3   4
 5   6   7   8    9   10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31

PS:
pprint means pretty print. A cool tool too.

Reference:
1 http://docs.python.org/lib/module-xmlrpclib.html
2 http://www-128.ibm.com/developerworks/library/ws-pyth10.html
3 http://groovy.codehaus.org/XMLRPC

python.cn上的几个大牛

limodou 人称李木头，此人技术深厚，擅长写作技术blog，信仰“自己的才是大家的”！

Zoom. Quiet 人称woodpecker社区大妈，虽称大妈，不过看上去像是位艺术家，发型极具特色
http://blog.zoomquiet.org/pyblosxom/
http://www.douban.com/people/zoomq/

刘鑫 March.Liu
http://blog.csdn.net/ccat

宁柯
beyking@gmail.com
www.china-django.com

zsp(张沈鹏)

Install apache 2.0.x, mysql 5.0.*, php 5.2.0 on Mac OS X

1 install apache
./configure
make
sudo make install

apache installed on /usr/local/apache2

2 install mysql
cp -Rf mysql-5.0.* /System/Library/Frameworks/Mysql.framework/
ln -s /System/Library/Frameworks/Mysql.framework/mysql-5.0.* /usr/local/mysql
ln -s /usr/local/mysql/data /Volumes/WRK/mysql-4.1.22/data

start mysql (do NOT start as root):
/usr/local/mysql/bin/mysqld
stop mysql
/usr/local/mysql/bin/mysqladmin shutdown

3 install php
./configure --with-apxs2=/usr/local/apache2/bin/apxs --with-mysql=/usr/local/mysql --prefix=/usr/local/php
make
sudo make install
cp src/php-5.2.0/php.ini-dist /usr/local/php/php.ini

4 config apache
vi /usr/local/apache2/conf/httpd.conf

LoadModule php5_module        modules/libphp5.so
AddType application/x-httpd-php .php .phtml
AddType application/x-httpd-php-source .phps
PHPIniDir "/usr/local/php"

DirectoryIndex index.html index.html.var index.php

ln -s /usr/local/apache2/htdocs /Volumes/WRK/Apache2/htdocs

that's all.

Gentoo vs Debian

A great article about gentoo and debian.
http://www.tomvergote.be/writings/Linux/Debian-Gentoo-production-environment.html

Recommend: firefox plugin,chmreader

url
https://addons.mozilla.org/firefox/3235/
It works fine under linux, parse chinese files correctly.

ps:
chmsee,gnochm,xchm all have bugs
libchm-bin recommended by others but I have not tried.

perfect python

 
 

file('new','w').writelines(set(file('old')))

easy, powerful!

Notes about shell.

1 [ -h file ], true if the file is a soft link. (guess from hadoop-deamon.sh)
2 echo $?, return the last command exit status.

code snippet from hadoop-deamon.sh

this="$0"
while [ -h "$this" ]; do
ls=`ls -ld "$this"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '.*/.*' > /dev/null; then
this="$link"
else
this=`dirname "$this"`/"$link"
fi
done

Tuning Performance in Weblogic on Solaris Sparc Box

Recently I worked on a j2ee project based on jsf, weblogic on solaris sparc box.
Developing on a windows box, it works fine. However when I deploy it onto the solaris 9, sparc box, the performance decline much.
the profiler shows that

1 invocation weblogic.servlet.internal.ServletStubImpl.getServlet() calls
1 invocation weblogic.servlet.internal.ServletStubImpl.checkForReload() calls
1 invocation weblogic.servlet.internal.ServletStubImpl.needToReload () calls
1 invocation weblogic.utils.classloaders.ChangeAwareClassLoader.upToDate() calls
2546 invocations weblogic.utils.classloaders.MultiClassFinder.getClassSource() calls
2546 invocations weblogic.utils.classloaders.MultiClassFinder.getSource () calls
2546 invocations weblogic.utils.classloaders.MultiClassFinder.getSource()

so about 70-80% of time is used by getServlet() calls. but on windows box, there is no such wired thing.
I found differences between windows and Solaris edition of ServletStubImpl.java decompiled from weblogic.jar.
Who knows why? but I have no time to inspect it.
From the source code, I know it checks whether the servlets are needed to reload and get the answer "yes", so it reloads thousands of servlets.
But it is running in production mode, weblogic should not check that. Maybe a bug?
The light throwed. I found another way to make weblogic never reload servlet files by searching Dev2Dev articles.
An excellent article http://dev2dev.bea.com/pub/a/2005/01/jsp_reloaded.html

Just set the value of servlet-reload-check-secs to -1!

<!DOCTYPE weblogic-web-app PUBLIC "-//BEA Systems, Inc.//DTD Web Application 8.1//EN"
 "http://www.bea.com/servers/wls810/dtd/weblogic810-web-jar.dtd">

<weblogic-web-app>
  <container-descriptor>
      <servlet-reload-check-secs>-1</servlet-reload-check-secs>
  </container-descriptor>
</weblogic-web-app>

And the article also describes how to make weblogic never recompile jsp and some other interesting things.

how to solve php "Warning: Cannot modify header information" in gallina

when using gallina, this warning always shows up:
Warning: Cannot modify header information - headers already sent by ......

change the encoding from gb2312 to utf-8 and the php works fine.

several ways to monitor the jvm

using jconsole in jdk1.5

• see http://java.sun.com/j2se/1.5.0/docs/guide/management/overview.html
• or http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
• add -Dcom.sun.management.jmxremote to vm paras
• start /bin/jconsole

Runtime.getRuntime().freeMemory() returns current freeMemory

using -Xloggc: para

• add '-Xloggc:gc.log' to vm parameters and the file gc.log shows detail memory usage from startup
• there is many tools to analyze the file to show you figures.

my first step to aspectJ

aspectJ is an extension to java. It is built on java and is a free walker through java's classes.