February 25, 2016

Celery giving "RestartFreqExceeded" with RabbitMQ

I recently tried setting up a new rabbitmq in an effort to migrate off of SQS and give back a little visibility into running tasks. SQS is opaque and lacks a lot of what I would consider basic features that rabbit has, and I figured I should do it the hard way and not use cloudamqp.com first.

As soon as I started up the worker, I began getting a stream of error messages:

[CRITICAL/MainProcess] Frequent restarts detected: RestartFreqExceeded('5 in 1s',)
Traceback (most recent call last):
  File "<venv>/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 284, in start
    self._restart_state.step()
  File "/usr/local/lib/python2.7/dist-packages/billiard/common.py", line 130, in step
    raise self.RestartFreqExceeded("%r in %rs" % (R, self.maxT))
RestartFreqExceeded: 5 in 1s

Googling was no help, even the specific exception name turned up nothing useful; mostly just source code references.

The solution turned out to be annoyingly simple, and came down to an incorrect broker url.

# wrong
BROKER_URL=amqp://devuser:devpassword@somequeue.io/dev-vhost/

# right!
BROKER_URL=amqp://devuser:devpassword@somequeue.io/dev-vhost

Not very wrong, just a tiny bit wrong and a horribly obscure error message. The key was just to remove the / from the vhost name.