The total file descriptors number of rabbitmq is 1024 by default. This value is not big enough obviously when your openstack cluster grows. If the file descriptor or socket descriptor reaches its limit, the rabbitmq server will close incoming connections. and you will see the error below:
AMQP server on controller1:5672 is unreachable: timed out. Trying again in 1 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 2 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 4 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 6 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 8 seconds. AMQP server controller1:5672 closed the connection. Check login credentials: Socket closed AMQP server on controller1:5672 is unreachable: timed out. Trying again in 12 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 14 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 16 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 18 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 20 seconds. AMQP server controller1:5672 closed the connection. Check login credentials: Socket closed
To check the current status of file descriptors used, you can run command ‘service rabbitmq-server status’
the result is
{file_descriptors,[{total_limit,3996},
{total_used,966},
{sockets_limit,3594},
{sockets_used,964}]},
from the result above, you can see i have adjusted the total limit of file descriptors to 4096, you can adjust it to the value that fits your environment. To change it, add following command in /etc/default/rabbitmq-server
ulimit -S -n 4096
and restart rabbitmq service.