Server Trouble

As you might have noticed, my server was down tonight. This is a quick status update on what's going on.

I'm having a few stability issues with the server for some time now. It seems to freeze for an unknown reason (about once every two months). This time however it was a bit worse: the disk was full. For some reason, lighttpd wrote 42GB to it's error log with the following line:

2009-08-18 06:25:59: (mod_fastcgi.c.2861) wait for fd at connection: 1023

This line occurs several thousand times per second. Google didn't tell me anything useful about it though.

After purging the log and rebooting the server, several MySQL tables were corrupted (a side effect of a full disk). Nice to know, the following command will attempt to repair all tables:

#> mysqlrepair -p<yourpass> --all-databases

Next problem: when starting lighty again I noticed it consuming a lot of RAM. Nearly 3GB and growing. Some googling around revealed this old thread which lead me to try disabling each of my sites step by step to see if any of my scripts might causing this. For now I disabled my rafinfo site which seems to fix the memory problem for now.

However the server is still experiencing quite a high load for a Tuesday morning and I'm not quite sure why. Traffic seems to be quite normal at least. MySQL seems to be quite busy currently with mytop showing about 300 queries per second (but with 98% cache efficiency).

Kernel and package data:

Linux gir 2.6.26-2-amd64 #1 SMP Fri Aug 14 07:12:04 UTC 2009 x86_64 GNU/Linux
ii  lighttpd                  1.4.19-5
ii  mysql-server              5.0.51a-24+lenny1
ii  php5-cgi                  5.2.6.dfsg.1-1+lenny3

All this of course while I'm supposed to work of course :-/

If you have any hint what I should look at or if you had similar problems, please let me know in the comments.

Tags:: splitbrain.org,; server,; lighttpd,; mysqld

Similar posts:

Posted on Tuesday, August the 18th 2009 (3 years ago).

Have a look at cat /proc/sys/fs/file-max
is is set to 1024?

if so the error message is saying you are running out of file descriptors.

echo "65535" > /proc/sys/fs/file-max
to increase the number temporarily and

If you want this new value to survive across reboots you can add it to /etc/sysctl.conf

# Maximum number of open files permited
fs.file-max = 65535

2009-08-18 13:50:42

Gavin

Gavin, good thinking. But:

# cat /proc/sys/fs/file-max
572963

That value seems to be insanely high though. But my /etc/sysctl.conf doesn't have anything about that parameter, so it seems to be the kernel default!?

2009-08-18 13:55:46

Andi

Try asking this at Server Fault ( http://serverfault.com/ )

2009-08-24 03:45:23

Henri

I am sure you ran acorsss this one in your Google venture ...

http://redmine.lighttpd.net/issues/1398

... but maybe not.

In case of your MySQL try running the Tuning Primer Script from ...

http://www.day32.com/MySQL/

... that usually gives me an idea where to have a look on my MySQL first.

2009-08-26 14:24:41

Steffen