Networking

Troubleshooting Websense as Proxy for site access

Joseph JenkinsLeave a comment

I recently had to troubleshoot a problem with a client going through Websense as a proxy and trying to gain access to a site. The site has at https://somesite.com:11001. Every time I would go to the site I would just get a “Page could not be displayed”. I then wen through and started troubleshooting from the Websense side and couldn’t see anything in the interface itself, so I went to the log server and then stopped the logging service and ran it from the commandline with just the client I was testing with. However this didn’t even show that there was a hit from the client. I then had to go to the next level and troubleshoot with a packet capture and Wireshark. Once I was able to capture the traffic I could see that Websense was returning an error that the browser wouldn’t display. The issue came down to using https on port 11001 which wasn’t allowed in the Content Gateway on the Websense appliance. Once I added that I was able to browse successfully to the site and have it show up in the log server.

So below I have summarized the steps for someone else needing to do this type of troubleshooting.

How to use the Websense testlogserver to troubleshoot problems and limit the information that is seen:

  1. Log into the logging server
  2. Stop the “Websense Log Server” service
  3. Go into the c:program files (x86)WebsenseWeb Securitybin folder and run the testlogserver.exe -onlyip (ip address you want to see)
  4. You can now surf the site from that machine and see what errors are showing up in the log server to help determine the problem.
  5. If you need to go to another level then run a packet capture from the machine using Websense as an explicit proxy in your browser. You can then limit the capture to just the Websense IP.
  6. Once you have gone to the site you can then look at the packet capture and search for “http contains (site you are going to)”.
  7. You should be able to then decode the http stream and see all of the headers and information returned. This should help you in troubleshooting the issue.

Mac, Python, paramiko, all in a days work

Joseph JenkinsLeave a comment

I am trying to learn Python as I think it will be good for my day job. I bought a couple of books, but I am someone that learns by doing. I found some good scripts out on the internet that I wanted to modify and make use of. However I am also a mac user and so I wanted to be able to run these scripts on my Mac so that when I wanted I could run them from where ever I might be. I do on occasion travel to sites and do some extra curricular activities that might require this ability. So the mac has Python pre installed, it’s version 2.7.5, which seemed sufficient for my needs and what I wanted to do. The script I wanted to play with needed the paramiko module. I was able to download it and extract from here:

https://github.com/paramiko/paramiko

That was easy, however to install it said if I had setuptools would be best. So I found this site:

https://pypi.python.org/pypi/setuptools#unix-including-mac-os-x-curl

And was able to find a command to download and install setuptools.
***Make sure you are root, you will have a much better time of it.***

curl https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py -o – | python

So that installed correctly, however when I went into python and did an “import paramiko” I was told I needed a crypto module. I then went out and found this:

https://pypi.python.org/pypi/pycrypto

Downloaded it and of course I couldn’t use setuptools for it, it needed to be built and then installed. So that required me to get Xcode 5.1 for the cc compiler and load that on my machine. That was straight forward enough. So after the Xcode install I then ran:

python setup.py build

But I was getting this error:

error: command ‘cc’ failed with exit status 1

Turns out there is an issue with Python and Xcode 5.1. The fix for that is to run the following before doing the build and install:

export CFLAGS=-Qunused-arguments
export CPPFLAGS=-Qunused-arguments

Once that is done you can then go into the pcrypto folder and run:

python setup.py build
python setup.py install

Now you have everything you need to use paramiko to ssh into a cisco device from a mac and run some commands or do whatever it is you want.

I did find one other thing that is needed and that was as part of the connect string for paramiko, I needed to specify “allow_agent=False,look_for_keys=False” as part of the string. If I didn’t then I was getting password errors on the cisco switch I was testing with.

ssh.connect(‘x.x.x.x’, username=’name’, password=’password’, allow_agent=False,look_for_keys=False)

All in all it was a very educational day and I think some hours well spent. I am now going to take my scripts and look to put everything into variables and also specify some lists so I can run it against multiple machines.

Using Mcafee Enterprise Security Manager to monitor Anyconnect Groups

Joseph JenkinsLeave a comment

We use Anyconnect for our Remote Access solution and one of the issues I have is with other admins not putting people into the right groups or not putting them into groups at all.  So then what happens is they get stuck into the DfltGrpPolicy, which is definitely not where I want them since that doesn’t have the customization for each of the different groups.

Since Mcafee Enterprise Security Manager is monitoring my ASA and all of the logs are going to it, building an alert to notify me when people are in this group shouldn’t be an issue.  Here is what we are going to do:

Identify the group, in this case I want to know when people get assigned the DfltGrpPolicy policy.

Now what we are going to do is build an alarm that fires when it sees DfltGrpPolicy in the object field.  Since no one should be pulling this group for any reason anyone that is in it is misconfigured and needs to be moved to the appropriate group.

1. Go to System Properties and then click on Alarms

2. Choose an assignee and give the alarm a name.

3. Choose, Type: Field Match, Field: Object, Value: DfltGrpPolicy, Select your device of your Cisco ASA.

4. Choose your actions, in my case I am going to have it email when it sees that.  I am also going to customize the template so it only sends me the relevant information, such as user misconfigured and group they are in.

5. I leave Escalation blank

6. Click Finish and you are done.

Now sit back and admire your work and you should now be alerted for people misconfigured.

Monitoring Domain Admin Password Changes with Nitro/Mcafee Enterprise Security

Joseph JenkinsLeave a comment

We had a requirement to monitor when Domain Admins changed their password so that we could pass our PCI Audit for this year.  Mostly the issue came out of the fact that a hacker could just as easily change a domain admin password as they could create a new account in domain admins.  I was already monitoring domain admins for new accounts getting added/removed, this just seemed like the next step.

I came up with this method to be able to monitor our Domain Controllers with Nitro and then send me a notification/report.

1. Create a Watchlist
call it whatever you want, but make it a destination user type and then choose an assignee.

2. Input all of your Domain Admins into the Watchlist

3. After creating the watchlist, then go into Alarms and create a new alarm.
Give it a name
choose an assignee

4. Go to the Condition tab and choose “specified event rate”, set the event count to “1”, and choose the time frame as 10 minutes.  This is needed to trigger the alarm when you need it to fire.

5. then click on the filter icon, for your query filter choose the following:
signature id: 43-211006270,43-211006280,43-263047230
destination user: WL:(name of watch lists)
Under select a device, choose all of your domain controllers.  We have two domains so we are monitoring several machines.

6. Now here is where the magic comes in and the best alerting you can setup.  Under actions, choose Generate Report and then click configure
Here you are going to create a new report that lists the userid field and any other information you want.  You will also choose your email recipients here so that you can be notified when a domain admins password has changed.  Since the parameters from the alarm will not be passed to the report you will need to choose a few things for the filter at the bottom to keep it to what you are looking for.
signature id: 43-211006270,43-211006280,43-263047230
destination user: WL:(name of watch lists)
set the time range to the last 10 minutes

7. click Finish, you now have an alarm that will be triggered when a domain admin changes their password.

Turn it off when you are done with it

Joseph JenkinsLeave a comment

I learned a lesson that I thought I knew and thought I actually exercised.

“Turn off whatever it is if you aren’t using it.”

I like most want to know what is going on with my network at all times.  I especially want to know if someone is making a change to a key piece of infrastructure, not to mention it’s nice to show to the auditors when they ask.  I have an alert setup on our Nitro appliance that notifies me when someone is making a change to our firewalls.  Important since this device is what I use for segregation on my network and to keep my credit card data safe.  Nitro does this by monitoring the syslog coming out of the firewall and looking for a particular message which relates to a signature ID.  When it sees signature id it then sends a message to me and the other firewall admin so that we can look at each other and say yup we made that change.

I was noticing though over the last couple of weeks that when I would make a change I wasn’t always getting notified.  I would get some alerts but not others.  So I started with looking at the Nitro appliance to see if it was having a problem.  As I was debugging it I noticed that it wasn’t getting all of the messages to be able to alert off of.  Information is missing and I needed to find it.

I then started looking at my syslog config on my ASA, here it is for reference:

logging enable
logging console informational
logging buffered debugging
logging trap informational
logging host inside x.x.x.x
no logging message 106015
no logging message 313001
no logging message 313008
no logging message 419002
no logging message 106023
no logging message 710003
no logging message 106100
no logging message 302014
no logging message 609002
no logging message 609001
no logging message 302018

Nothing out of the ordinary or so I thought.  As I was using Google to look at stuff I came across some messages about the logging queue limit, which by default is 512.  I decided to look at that and see if that could be causing my issue.  When I looked I saw this:

sh logging queue

Logging Queue length limit : 512 msg(s)
-1123742434 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 512 msg on queue, 512 msgs most on queue

Definitely not good I am dropping messages and the queue is full.  I thought well may be I can increase the queue and everything will be fine after that.  So I did the following:

logging queue 1024
sh logging queue

Logging Queue length limit : 1024 msg(s)
-1123731334 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 1024 msg on queue, 1024 msgs most on queue

Immediately the queue jumped up and I was still dropping messages.

I thought okay may be I need to prune out the message I am logging on.  These are minor messages and I hadn’t needed this data in any of my investigations so I decided to kill it.

no logging message 302017
no logging message 302016
no logging message 302021
no logging message 302020

sh logging queue

Current 1024 msg on queue, 1024 msgs most on queue

No change and I am still dropping messages.  I then thought heck, may be I just need a bigger queue, that always solves the problem right.  Bigger is better:

logging queue 4196
sh logging queue

Logging Queue length limit : 4196 msg(s)
-1123742651 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 4196 msg on queue, 4196 msgs most on queue

That didn’t help out so much.  Immediately I am at a full queue and still dropping messages.  Then I looked back through my config again:

sh run logging

logging enable
logging console informational
logging buffered debugging
logging trap informational
logging queue 4196
logging host inside  x.x.x.x
no logging message 106015
no logging message 313001
no logging message 313008
no logging message 419002
no logging message 106023
no logging message 710003
no logging message 106100
no logging message 302014
no logging message 609002
no logging message 609001
no logging message 302018
no logging message 302017
no logging message 302016
no logging message 302021
no logging message 302020

Hmm, I wonder if logging to the console and buffer are causing my issue.  I am not using them currently and the last time I was troubleshooting I did turn them on.  Could I really have not cleaned up after myself and could this be causing me an issue?  I then did the following:

no logging console
no logging buffered
sh logging queue

Logging Queue length limit : 4196 msg(s)
-1123728024 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 0 msg on queue, 4196 msgs most on queue

Immediately the queue dropped down and there was nothing in it.  I then moved the queue back down to a smaller number.

logging queue 1024
sh logging queue

Logging Queue length limit : 1024 msg(s)
-1123728024 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 0 msg on queue, 4196 msgs most on queue

No messages on queue and no dropped messages.  Also all of my test alerts are now working correctly and everything seems to be fine.

Lesson relearned, when you turn on something make sure you turn it off.  Even if at the time you don’t think it will cause you an issue it may come and cause you an issue later.

Dynamic Arp Inspection/DHCP Snooping

Joseph JenkinsLeave a comment

In trying to remove Man in the Middle attacks for my network I started looking at Dynamic Arp Inspection(DAI) and DHCP Snooping.  I brought them home to my lab and started to play.  Here is what I read up on to figure out what to do and how to implement:

Research:
http://www.cisco.com/en/US/docs/switches/lan/catalyst3750x_3560x/software/release/12.2_58_se/configuration/guide/swdynarp.html#wp1039773
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/dynarp.html
http://packetlife.net/blog/2010/aug/18/dhcp-snooping-and-dynamic-arp-inspection/

How to configure:

First turn on DHCP SNOOPING on the switch or switches:

“ip dhcp snooping”
“ip dhcp snooping vlan (vlan to monitor)”

Let it run for a while and populate the dhcp snooping binding database.  This database is extremely important as only the bindings in here will be allowed to arp on the network.

To view the database you can use the following:

“show ip dhcp snooping binding”

Uplink ports and ports that will have dhcp servers will need to have the following put onto the interfaces or you won’t be able to get addresses:

“ip dhcp snooping trust”

Before you turn on Dynamic Arp Inspection you need to track down any dumb switches(switches that don’t support DAI) or hosts with a static IP address.

I recommend removing any dumb switches from the network as they just create security holes and will cause you nothing but problems.

Here is one solution, but I think it is better to get rid of them and easier to deal with:

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/dynarp.html#wp1077449

As for the static hosts you can do “ip arp inspection trust”, however I think a better method is to create a static ip address to arp address binding with the following:

“ip source binding (mac add) vlan (vlan to monitor) (ip address) interface (interface of host)”

This way someone can’t just remove the static host and take over their ip address.  Another option would be to change all hosts at the Distribution/Access layer to DHCP and put static bindings into the DHCP server for them to ensure their ip addresses don’t change.

To turn on Dynamic Arp Inspection

Identify your uplink ports and use the following command on them:

“ip arp inspection trust”

 When ready to turn on DAI then run:

“ip arp inspection vlan (vlan to monitor)”

By default all ports are untrusted and should have 1 host to 1 network port.

If you have more than one host to a port with a dumb switch you need to use “ip arp inspection trust”, or else the switch will drop all of the hosts on that port.

Switch Replacements

Joseph JenkinsLeave a comment

My company is still in the stone age in some ways, we have the latest and greatest in some things.  However when it comes to our switching environment someone made the foolish decision of putting 2’x2′ holes into the concrete and then mounting switches into the floor.  This idea was stupid as hell, because unfortunately there are very few switches that we can use in these holes.  So when I have a chance to actually replace switches with something that works better and isn’t ready to be decommissioned I am happy.  This was one of those weekends where I got to replace some of our old 3524 switches and bring us into the 2013 with some nice and shiny new 3750x stack switches.  The only downside of this upgrade was that I had to do it 30 feet in the air on top of one of our conference buildings.  So getting up there required the use of a lift and getting myself extremely dirty while digging through a bunch of equipment.

The upgrade took about 4 hours, mostly due to balancing and trying to not fall through the roof into the conference room below.  Once they were mounted and connected back to our core the connections came up and started working well.  I know that there are some that don’t like the 3750x stack switches, but I have to say I have had very good luck with them and they all have worked very well for me.  Plus the additional advantage is that I only have to manage one switch and I got to replace two other management headaches.

I really wish I could find a good solution though for the switches that are sitting in the floor, there are so many problems with them that I am afraid I am going to run out of 3524 replacements to go in there.

rx/tx load cheatsheet

Joseph JenkinsLeave a comment

During the holiday season this year I found myself in need of looking at the bandwidth metrics and needed to figure out load on the fly. Rather than do the math and possibly make a mistake I decided to use excel to my advantage and figure it out ahead of time. So taking this:

reliability 255/255, txload 201/255, rxload 49/255

And then compare it to this:

rx/tx Total Percentage
255 255 100.00%
254 255 99.61%
253 255 99.22%
252 255 98.82%
251 255 98.43%
250 255 98.04%
249 255 97.65%
248 255 97.25%
247 255 96.86%
246 255 96.47%
245 255 96.08%
244 255 95.69%
243 255 95.29%
242 255 94.90%
241 255 94.51%
240 255 94.12%
239 255 93.73%
238 255 93.33%
237 255 92.94%
236 255 92.55%
235 255 92.16%
234 255 91.76%
233 255 91.37%
232 255 90.98%
231 255 90.59%
230 255 90.20%
229 255 89.80%
228 255 89.41%
227 255 89.02%
226 255 88.63%
225 255 88.24%
224 255 87.84%
223 255 87.45%
222 255 87.06%
221 255 86.67%
220 255 86.27%
219 255 85.88%
218 255 85.49%
217 255 85.10%
216 255 84.71%
215 255 84.31%
214 255 83.92%
213 255 83.53%
212 255 83.14%
211 255 82.75%
210 255 82.35%
209 255 81.96%
208 255 81.57%
207 255 81.18%
206 255 80.78%
205 255 80.39%
204 255 80.00%
203 255 79.61%
202 255 79.22%
201 255 78.82%
200 255 78.43%
199 255 78.04%
198 255 77.65%
197 255 77.25%
196 255 76.86%
195 255 76.47%
194 255 76.08%
193 255 75.69%
192 255 75.29%
191 255 74.90%
190 255 74.51%
189 255 74.12%
188 255 73.73%
187 255 73.33%
186 255 72.94%
185 255 72.55%
184 255 72.16%
183 255 71.76%
182 255 71.37%
181 255 70.98%
180 255 70.59%
179 255 70.20%
178 255 69.80%
177 255 69.41%
176 255 69.02%
175 255 68.63%
174 255 68.24%
173 255 67.84%
172 255 67.45%
171 255 67.06%
170 255 66.67%
169 255 66.27%
168 255 65.88%
167 255 65.49%
166 255 65.10%
165 255 64.71%
164 255 64.31%
163 255 63.92%
162 255 63.53%
161 255 63.14%
160 255 62.75%
159 255 62.35%
158 255 61.96%
157 255 61.57%
156 255 61.18%
155 255 60.78%
154 255 60.39%
153 255 60.00%
152 255 59.61%
151 255 59.22%
150 255 58.82%
149 255 58.43%
148 255 58.04%
147 255 57.65%
146 255 57.25%
145 255 56.86%
144 255 56.47%
143 255 56.08%
142 255 55.69%
141 255 55.29%
140 255 54.90%
139 255 54.51%
138 255 54.12%
137 255 53.73%
136 255 53.33%
135 255 52.94%
134 255 52.55%
133 255 52.16%
132 255 51.76%
131 255 51.37%
130 255 50.98%
129 255 50.59%
128 255 50.20%
127 255 49.80%
126 255 49.41%
125 255 49.02%
124 255 48.63%
123 255 48.24%
122 255 47.84%
121 255 47.45%
120 255 47.06%
119 255 46.67%
118 255 46.27%
117 255 45.88%
116 255 45.49%
115 255 45.10%
114 255 44.71%
113 255 44.31%
112 255 43.92%
111 255 43.53%
110 255 43.14%
109 255 42.75%
108 255 42.35%
107 255 41.96%
106 255 41.57%
105 255 41.18%
104 255 40.78%
103 255 40.39%
102 255 40.00%
101 255 39.61%
100 255 39.22%
99 255 38.82%
98 255 38.43%
97 255 38.04%
96 255 37.65%
95 255 37.25%
94 255 36.86%
93 255 36.47%
92 255 36.08%
91 255 35.69%
90 255 35.29%
89 255 34.90%
88 255 34.51%
87 255 34.12%
86 255 33.73%
85 255 33.33%
84 255 32.94%
83 255 32.55%
82 255 32.16%
81 255 31.76%
80 255 31.37%
79 255 30.98%
78 255 30.59%
77 255 30.20%
76 255 29.80%
75 255 29.41%
74 255 29.02%
73 255 28.63%
72 255 28.24%
71 255 27.84%
70 255 27.45%
69 255 27.06%
68 255 26.67%
67 255 26.27%
66 255 25.88%
65 255 25.49%
64 255 25.10%
63 255 24.71%
62 255 24.31%
61 255 23.92%
60 255 23.53%
59 255 23.14%
58 255 22.75%
57 255 22.35%
56 255 21.96%
55 255 21.57%
54 255 21.18%
53 255 20.78%
52 255 20.39%
51 255 20.00%
50 255 19.61%
49 255 19.22%
48 255 18.82%
47 255 18.43%
46 255 18.04%
45 255 17.65%
44 255 17.25%
43 255 16.86%
42 255 16.47%
41 255 16.08%
40 255 15.69%
39 255 15.29%
38 255 14.90%
37 255 14.51%
36 255 14.12%
35 255 13.73%
34 255 13.33%
33 255 12.94%
32 255 12.55%
31 255 12.16%
30 255 11.76%
29 255 11.37%
28 255 10.98%
27 255 10.59%
26 255 10.20%
25 255 9.80%
24 255 9.41%
23 255 9.02%
22 255 8.63%
21 255 8.24%
20 255 7.84%
19 255 7.45%
18 255 7.06%
17 255 6.67%
16 255 6.27%
15 255 5.88%
14 255 5.49%
13 255 5.10%
12 255 4.71%
11 255 4.31%
10 255 3.92%
9 255 3.53%
8 255 3.14%
7 255 2.75%
6 255 2.35%
5 255 1.96%
4 255 1.57%
3 255 1.18%
2 255 0.78%
1 255 0.39%

Tells me load is about 79%, fast easy and not likely to get messed up.

Useful UCS Links

Joseph JenkinsLeave a comment

I have been in the process of upgrading our UCS to firmware version 2.0.3b. While I read through the Cisco doc I still didn’t have a great feel for what the upgrade would entail. I did some searching and found the following two links which I think lay out the process really well and make it easy to understand along with some concepts that aren’t as straight forward as I thought they could be.

Actual upgrade of a chassis:
http://terenceluk.blogspot.com/2011/10/updating-cisco-ucs-b-series.html

Difference between Activate and Update on the firmware management tab.

Cisco UCS Firmware Update vs Activate

I hope this help someone else out and make life easier for them.