To fix this problem (and after it gave me a lot of headaches and stomachache :P), I gave Slackware a go. Now after 3 days, I never notice a single hang and no slow whatsoever. This brings to happy users and the most important a happy system admin :-) . You can try access my webmail here.
Happy New Year!