Since yesterday afternoon, the Jabber server seems to work like a charm, let's hope it'll continue to do so. Sorry for any inconvenience caused by the stability problems. Please report any ongoing problems via the contact form.
What happened? Well, Wippien is a P2P VPN software that is based on XMPP. On jabber.rueckgr.at there a few users registered that regularly use that software. It seems that Wippien exchanges a lot of messages. If a recipient goes offline while receiving such a bunch of messages, they are stored in the database until the recipient comes back online. That works perfectly as long as the number of messages waiting for a certain user does not exceed a certain amount. However, one user had about 26.000 messages waiting to be delivered. Whenever that user came online, ejabberd tried to fetch and deliver these messages. When trying to do so, ejabberd consumed lots of CPU time and allocated more and more memory, but the messages were still not delivered. As soon ejabberd had allocated about 4-5 GB of memory, it either crashed or was terminated by Linux' out-of-memory killer.
Unfortunately, it took quite a time to find out that the large number of offline messages waiting to be delivered was the reason for ejabberd being unstable. As soon as these messages were removed, everything started to work smoothly again.
To avoid similar problems in the future, the following countermeasures will be applied:
- Offline messages older than 60 days will be deleted periodically.
- If the number of offline messages of a certain user exceeds 500, the oldest messages will be deleted until the number of offline messages drops to 500.
These countermeasures are subject to change if it turns out that they need to be readjusted to ensure stable operation of Jabber services.
Deutsch
English