Archive for December, 2011

Too many open files on Tomcat

Posted by smoreau in Java on 20 Dec 2011

The other day, one of my websites was not available anymore. Looking at the log files, I found the following exception:

Dec 7, 2011 1:22:39 AM org.apache.jk.common.ChannelSocket acceptConnections
WARNING: Exception executing accept
java.net.SocketException: Too many open files
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
	at java.net.ServerSocket.implAccept(ServerSocket.java:450)
	at java.net.ServerSocket.accept(ServerSocket.java:421)
	at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:307)
	at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:661)
	at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:872)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
	at java.lang.Thread.run(Thread.java:595)

This was the first time I was getting this exception. What is even stranger is that I didn’t change anything on this application for quite a while!
Anyway, I first did what I usually do with Tomcat: restart it! This fixed the issue but only for a few hours before it crashed again.

After some investigation, it seems that Tomcat was reaching the limit of open file descriptors allowed in this machine (1024 in my case).
To get the maximum number of open file descriptors, simply type the following command:

ulimit -n

It is possible to increase this value by editing the file /etc/security/limits.conf and adding the new limit for the user running Tomcat. However, this is not recommended as 1024 should be sufficient.

The second thing I did was to check the list of open files used by the Tomcat process:

lsof -p

What I found by running this command was a bit odd. It seems that Tomcat was having a multitude of opened connections to one of the web services used by the application. So it looks like the connections between my website and the web service were never closed!  😐
Because I didn’t change the code on my side, I asked the third party who owns the web service to check their code. I don’t know what was the root cause of the problem but they fixed it on their side and it is now working fine.

In conclusion, if you get the same exception, try to find where the problem is coming from before increasing the maximum number of open file descriptors. 😉

, , , , , ,

No Comments

Samba access problem with Mac OS X 10.6+

Posted by smoreau in Linux on 13 Dec 2011

This is a problem I encountered when I upgraded Mac OS X from the version 10.5 (Leopard) to 10.6 (Snow Leopard). One of my friend also got a similar problem when she upgraded to the version 10.7 (Lion).
This issue was affecting the access to the network shares set up with Samba (version 3.0.24) on my D-Link DNS-323. For some reason, I wasn’t able to authenticate on the shares as soon as I upgraded to Snow Leopard!

Here is the error message I was getting:

After browsing a few forums on the web, I finally found a solution. 🙂
I simply had to change the security mode in the Samba configuration file (smb.conf) to read:

security = USER

Note that this property can be found under the [global] section.

For more information about the Samba security mode, please read the following article by Jack Wallen:
Understanding Samba security modes

, , , , ,

No Comments

S3 command failed if the time is not synced

Posted by smoreau in Linux on 06 Dec 2011

This is already the second post about the s3sync ruby program. The first article was focused on monitoring s3sync with Zabbix.

I will talk on this one about an error I got when running the S3 synchronisation:

S3 command failed:
list_bucket prefix /data max-keys 200 delimiter /
With result 403 Forbidden
S3 ERROR: #
s3sync.rb:290:in `+': can't convert nil into Array (TypeError)
	from s3sync.rb:290:in `s3TreeRecurse'
	from s3sync.rb:346:in `main'
	from ./thread_generator.rb:79:in `call'
	from ./thread_generator.rb:79:in `initialize'
	from ./thread_generator.rb:76:in `new'
	from ./thread_generator.rb:76:in `initialize'
	from s3sync.rb:267:in `new'
	from s3sync.rb:267:in `main'
	from s3sync.rb:735

As you can see, this error is not very human-friendly! 😮 The only thing we know is that the S3 command failed because of the error can't convert nil into Array. It looks to me like an internal error within s3sync…

But after some investigation, it appears it is simply because the system date on the server is not correct. I cannot tell you how much time I spent on this one!  😯

Anyway, if you are doing automatic backups as describe on John Eberly’s blog, you need to add the following code at the top of your upload.sh script:

# update the system date
/usr/sbin/ntpdate 3.uk.pool.ntp.org 2.uk.pool.ntp.org 1.uk.pool.ntp.org 0.uk.pool.ntp.org

NB: please find below the command lines I use to install ntpdate on a Debian server:

apt-get install ntpdate
dpkg-reconfigure tzdata

, , , , , , ,

2 Comments