I learned a lesson that I thought I knew and thought I actually exercised.

“Turn off whatever it is if you aren’t using it.”

I like most want to know what is going on with my network at all times.  I especially want to know if someone is making a change to a key piece of infrastructure, not to mention it’s nice to show to the auditors when they ask.  I have an alert setup on our Nitro appliance that notifies me when someone is making a change to our firewalls.  Important since this device is what I use for segregation on my network and to keep my credit card data safe.  Nitro does this by monitoring the syslog coming out of the firewall and looking for a particular message which relates to a signature ID.  When it sees signature id it then sends a message to me and the other firewall admin so that we can look at each other and say yup we made that change.

I was noticing though over the last couple of weeks that when I would make a change I wasn’t always getting notified.  I would get some alerts but not others.  So I started with looking at the Nitro appliance to see if it was having a problem.  As I was debugging it I noticed that it wasn’t getting all of the messages to be able to alert off of.  Information is missing and I needed to find it.

I then started looking at my syslog config on my ASA, here it is for reference:

logging enable
logging console informational
logging buffered debugging
logging trap informational
logging host inside x.x.x.x
no logging message 106015
no logging message 313001
no logging message 313008
no logging message 419002
no logging message 106023
no logging message 710003
no logging message 106100
no logging message 302014
no logging message 609002
no logging message 609001
no logging message 302018

Nothing out of the ordinary or so I thought.  As I was using Google to look at stuff I came across some messages about the logging queue limit, which by default is 512.  I decided to look at that and see if that could be causing my issue.  When I looked I saw this:

sh logging queue

Logging Queue length limit : 512 msg(s)
-1123742434 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 512 msg on queue, 512 msgs most on queue

Definitely not good I am dropping messages and the queue is full.  I thought well may be I can increase the queue and everything will be fine after that.  So I did the following:

logging queue 1024
sh logging queue

Logging Queue length limit : 1024 msg(s)
-1123731334 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 1024 msg on queue, 1024 msgs most on queue

Immediately the queue jumped up and I was still dropping messages.

I thought okay may be I need to prune out the message I am logging on.  These are minor messages and I hadn’t needed this data in any of my investigations so I decided to kill it.

no logging message 302017
no logging message 302016
no logging message 302021
no logging message 302020

sh logging queue

Current 1024 msg on queue, 1024 msgs most on queue

No change and I am still dropping messages.  I then thought heck, may be I just need a bigger queue, that always solves the problem right.  Bigger is better:

logging queue 4196
sh logging queue

Logging Queue length limit : 4196 msg(s)
-1123742651 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 4196 msg on queue, 4196 msgs most on queue

That didn’t help out so much.  Immediately I am at a full queue and still dropping messages.  Then I looked back through my config again:

sh run logging

logging enable
logging console informational
logging buffered debugging
logging trap informational
logging queue 4196
logging host inside  x.x.x.x
no logging message 106015
no logging message 313001
no logging message 313008
no logging message 419002
no logging message 106023
no logging message 710003
no logging message 106100
no logging message 302014
no logging message 609002
no logging message 609001
no logging message 302018
no logging message 302017
no logging message 302016
no logging message 302021
no logging message 302020

Hmm, I wonder if logging to the console and buffer are causing my issue.  I am not using them currently and the last time I was troubleshooting I did turn them on.  Could I really have not cleaned up after myself and could this be causing me an issue?  I then did the following:

no logging console
no logging buffered
sh logging queue

Logging Queue length limit : 4196 msg(s)
-1123728024 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 0 msg on queue, 4196 msgs most on queue

Immediately the queue dropped down and there was nothing in it.  I then moved the queue back down to a smaller number.

logging queue 1024
sh logging queue

Logging Queue length limit : 1024 msg(s)
-1123728024 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 0 msg on queue, 4196 msgs most on queue

No messages on queue and no dropped messages.  Also all of my test alerts are now working correctly and everything seems to be fine.

Lesson relearned, when you turn on something make sure you turn it off.  Even if at the time you don’t think it will cause you an issue it may come and cause you an issue later.