Security

Check your Raid: Cisco ASA and Sourcefire

Joseph JenkinsLeave a comment

Over the past week I had an issue where one of my Cisco 5545 with a Sourcefire module went down and failed and I couldn’t get it restarted. When I looked at the console for the SFR Module I saw disk errors and I opened a ticket with Cisco to have them look at it. One thing that I found appalling was the quality of Cisco TAC engineers has dropped dramatically. I spent more time on the phone with these guys not knowing what to do and and my showing them commands that I had just googled and what needed to be done. If these guys are supposed to be the experts in the device and technology I am not impressed. Especially since Cisco keeps raising my rates and the quality seems to get lower rather than better.

Back to the issue:

The Cisco 5545 Sourcefire unit has two SSDs in a Raid 1 configuration, so you would think that if one failed the other would take over. At least that is what I thought, however it turns out that both of the SSDs had failed and there was no notification at all on the unit itself or in the logs as to one of the units being bad, let alone both of them. The only place I found it was running the “sh raid” command on the terminal. After seeing the failure of this unit, I then went through the rest of my 5545s with Sourcefire modules and found two others that had a failed drive and there was no warning, no error lights on the drive or the firewall itself. I had to run the command to find the issue.

Here is what a healthy raid set looks like:

/dev/md0:
Version : 1.2
Creation Time : Fri Feb 19 18:27:16 2021
Raid Level : raid1
Array Size : 124969216 (119.18 GiB 127.97 GB)
Used Dev Size : 124969216 (119.18 GiB 127.97 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Wed Jun  2 20:05:01 2021
      State : clean

Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

   Name : ciscoasa:0  (local to host ciscoasa)
   UUID : 244baa9a:b6e40506:f7384510:fcb42706
 Events : 12123

Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
2 8 16 1 active sync /dev/sdb

Here’s what an unhealthy raid set looks like:

/dev/md0:
        Version : 1.2
  Creation Time : Mon May 25 12:42:13 2020
     Raid Level : raid1
     Array Size : 124969216 (119.18 GiB 127.97 GB)
  Used Dev Size : 124969216 (119.18 GiB 127.97 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Jun  2 20:01:32 2021
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0

           Name : ciscoasa:0  (local to host ciscoasa)
           UUID : 0ed2ca7c:260897dd:f183f4bf:c0f15bfb
         Events : 12258234

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       2       0        0        2      removed

       2       8       16        -      faulty   /dev/sdb

I can’t believe there is no logs or notifications, Solarwinds didn’t pick up the hardware issues. You would think there would be some sort of notification sent out or the HD light on the firewall would turn orange, what a novel concept to notify people of a failed hardware item before it causes major problems.

So if you run any of the modules in your ASA firewalls, make sure to check the raid level and that the drives are in a healthy state, if not get the ticket open with TAC. Where they can give you such brilliant ideas as move the faulty drive to another ASA, or swap the drives(which causes the firewall to crash, so don’t do it).

Microsoft MFA login with Fortigate and Forticlient for SSLVPN

Joseph Jenkins9 Comments

Since I am tired of being a beta tester for Cisco products. I decided to try a different firewall this time around for my company. I looked at both Fortigate and Palo Alto as they seemed to be the leaders in the market right now. I did a bake off for features/functionality vs cost and Fortigate came out as the winner. The firewall was implemented with minimal issues and has been working flawlessly for us. While we were on this project we are also in the process of moving to Azure AD so I decided to combine the Microsoft MFA with our new firewall/vpn solution to save ourselves some money since then we wouldn’t need another 2 factor solution.

I went through the documentation from Fortigate and Microsoft on setting up the SAML authentication and it was pretty good for the most part. Here was the main document that I followed to get everything setup:
https://docs.microsoft.com/en-us/azure/active-directory/saas-apps/fortigate-ssl-vpn-tutorial
I did run into a few issues that I had to fix to get everything working with group memberships, so that users would be enabled to login based on their group and would have the correct policy applied to them.

Here are some things to be aware of and the changes I needed to make:

1. You must be on the 6.4.x code for Fortigate. There are issues with the lower code versions and SAML not working correctly or populating the tables with the necessary information.
2. Wipe out all of the extra entries under Users and Attributes Claims in Azure AD. This is all you should have:
UntitledImage
3. Here is the necessary configuration on the Fortigate side:
config user saml
edit “azure”
set cert “Fortinet_Factory”
set entity-id “https://XXXXXXX/remote/saml/metadata”
set single-sign-on-url “https://XXXXXX/remote/saml/login”
set single-logout-url “https://SSSSSSSS/remote/saml/logout”
set idp-entity-id “https://sts.windows.net/6XXXXXXX/”
set idp-single-sign-on-url “https://login.microsoftonline.com/XXXXX/saml2”
set idp-single-logout-url “https://login.microsoftonline.com/common/wsfederation?wa=wsignout1.0”
set idp-cert “REMOTE_Cert_2”
set user-name “username”
set group-name “group”
next
end

After these changes everything worked perfectly, I am now in the process of rolling out our new vpn to the users in the company along with the Microsoft MFA client.

Setting up Brocade Switches to do Tacacs+ authentication with Cisco ACS

Joseph JenkinsLeave a comment

This was a fun one, I had some issues with getting my Brocade switches to continue doing Radius auth with my Cisco ACS so I switched to TACACS+ for them. Had a few issues, but was able to piece this configuration together and have it work correctly.

First setup your servers on the Brocade Side, I find it easiest to work on the command line and define everything that is needed.

aaaconfig –add (first ip) -conf tacacs+ -p 49 -s (secret) -t 5 -a pap
aaaconfig –add (second ip) -conf tacacs+ -p 49 -s (secret) -t 5 -a pap
aaaconfig –authspec tacacs+

The important thing to note here is to use PAP for authentication if you are doing this against Active Directory. AD will not support CHAP which is the default on the Brocade Switches.

Once that is setup you now have no access into your Brocade devices because they are doing TACACS+ authentication and you haven’t defined them on the ACS server yet. Lets take care of that part.

Log into ACS:

Navigate to Device Administration/Shell Profiles, Create a new one, I called mine BROCADE-TAC. Once created click on the “Custom Attributes” and put in the following:shell profileThis will ensure that when you log in, you will log in as an admin on the Switch.

Once this is done go to your Access Polices/Device Admin or whatever you have it called that does your Tacacs Rule authentication. Create a new line in there and then choose your Active Directory login/groupsauth policyThen under Shell Profile, choose the one that you created before. Now depending on your environment you may want to define a specific command set. In my case I am just using Permit All, but you can create different sets for different users depending on what you are trying to limit and who has access to your device. So if you had operators you could permit the show commands or some limited feature set for a junior admin. I only have a few people that log in and they all needed to be full admins, so this was the best choice for me. Mainly we wanted to make sure that we had auditing turned on and would know when someone was logged in making changes to the devices.

 

 

Mac, Python, paramiko, all in a days work

Joseph JenkinsLeave a comment

I am trying to learn Python as I think it will be good for my day job. I bought a couple of books, but I am someone that learns by doing. I found some good scripts out on the internet that I wanted to modify and make use of. However I am also a mac user and so I wanted to be able to run these scripts on my Mac so that when I wanted I could run them from where ever I might be. I do on occasion travel to sites and do some extra curricular activities that might require this ability. So the mac has Python pre installed, it’s version 2.7.5, which seemed sufficient for my needs and what I wanted to do. The script I wanted to play with needed the paramiko module. I was able to download it and extract from here:

https://github.com/paramiko/paramiko

That was easy, however to install it said if I had setuptools would be best. So I found this site:

https://pypi.python.org/pypi/setuptools#unix-including-mac-os-x-curl

And was able to find a command to download and install setuptools.
***Make sure you are root, you will have a much better time of it.***

curl https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py -o – | python

So that installed correctly, however when I went into python and did an “import paramiko” I was told I needed a crypto module. I then went out and found this:

https://pypi.python.org/pypi/pycrypto

Downloaded it and of course I couldn’t use setuptools for it, it needed to be built and then installed. So that required me to get Xcode 5.1 for the cc compiler and load that on my machine. That was straight forward enough. So after the Xcode install I then ran:

python setup.py build

But I was getting this error:

error: command ‘cc’ failed with exit status 1

Turns out there is an issue with Python and Xcode 5.1. The fix for that is to run the following before doing the build and install:

export CFLAGS=-Qunused-arguments
export CPPFLAGS=-Qunused-arguments

Once that is done you can then go into the pcrypto folder and run:

python setup.py build
python setup.py install

Now you have everything you need to use paramiko to ssh into a cisco device from a mac and run some commands or do whatever it is you want.

I did find one other thing that is needed and that was as part of the connect string for paramiko, I needed to specify “allow_agent=False,look_for_keys=False” as part of the string. If I didn’t then I was getting password errors on the cisco switch I was testing with.

ssh.connect(‘x.x.x.x’, username=’name’, password=’password’, allow_agent=False,look_for_keys=False)

All in all it was a very educational day and I think some hours well spent. I am now going to take my scripts and look to put everything into variables and also specify some lists so I can run it against multiple machines.

Using Mcafee Enterprise Security Manager to monitor Anyconnect Groups

Joseph JenkinsLeave a comment

We use Anyconnect for our Remote Access solution and one of the issues I have is with other admins not putting people into the right groups or not putting them into groups at all.  So then what happens is they get stuck into the DfltGrpPolicy, which is definitely not where I want them since that doesn’t have the customization for each of the different groups.

Since Mcafee Enterprise Security Manager is monitoring my ASA and all of the logs are going to it, building an alert to notify me when people are in this group shouldn’t be an issue.  Here is what we are going to do:

Identify the group, in this case I want to know when people get assigned the DfltGrpPolicy policy.

Now what we are going to do is build an alarm that fires when it sees DfltGrpPolicy in the object field.  Since no one should be pulling this group for any reason anyone that is in it is misconfigured and needs to be moved to the appropriate group.

1. Go to System Properties and then click on Alarms

2. Choose an assignee and give the alarm a name.

3. Choose, Type: Field Match, Field: Object, Value: DfltGrpPolicy, Select your device of your Cisco ASA.

4. Choose your actions, in my case I am going to have it email when it sees that.  I am also going to customize the template so it only sends me the relevant information, such as user misconfigured and group they are in.

5. I leave Escalation blank

6. Click Finish and you are done.

Now sit back and admire your work and you should now be alerted for people misconfigured.

Monitoring Domain Admin Password Changes with Nitro/Mcafee Enterprise Security

Joseph JenkinsLeave a comment

We had a requirement to monitor when Domain Admins changed their password so that we could pass our PCI Audit for this year.  Mostly the issue came out of the fact that a hacker could just as easily change a domain admin password as they could create a new account in domain admins.  I was already monitoring domain admins for new accounts getting added/removed, this just seemed like the next step.

I came up with this method to be able to monitor our Domain Controllers with Nitro and then send me a notification/report.

1. Create a Watchlist
call it whatever you want, but make it a destination user type and then choose an assignee.

2. Input all of your Domain Admins into the Watchlist

3. After creating the watchlist, then go into Alarms and create a new alarm.
Give it a name
choose an assignee

4. Go to the Condition tab and choose “specified event rate”, set the event count to “1”, and choose the time frame as 10 minutes.  This is needed to trigger the alarm when you need it to fire.

5. then click on the filter icon, for your query filter choose the following:
signature id: 43-211006270,43-211006280,43-263047230
destination user: WL:(name of watch lists)
Under select a device, choose all of your domain controllers.  We have two domains so we are monitoring several machines.

6. Now here is where the magic comes in and the best alerting you can setup.  Under actions, choose Generate Report and then click configure
Here you are going to create a new report that lists the userid field and any other information you want.  You will also choose your email recipients here so that you can be notified when a domain admins password has changed.  Since the parameters from the alarm will not be passed to the report you will need to choose a few things for the filter at the bottom to keep it to what you are looking for.
signature id: 43-211006270,43-211006280,43-263047230
destination user: WL:(name of watch lists)
set the time range to the last 10 minutes

7. click Finish, you now have an alarm that will be triggered when a domain admin changes their password.

Turn it off when you are done with it

Joseph JenkinsLeave a comment

I learned a lesson that I thought I knew and thought I actually exercised.

“Turn off whatever it is if you aren’t using it.”

I like most want to know what is going on with my network at all times.  I especially want to know if someone is making a change to a key piece of infrastructure, not to mention it’s nice to show to the auditors when they ask.  I have an alert setup on our Nitro appliance that notifies me when someone is making a change to our firewalls.  Important since this device is what I use for segregation on my network and to keep my credit card data safe.  Nitro does this by monitoring the syslog coming out of the firewall and looking for a particular message which relates to a signature ID.  When it sees signature id it then sends a message to me and the other firewall admin so that we can look at each other and say yup we made that change.

I was noticing though over the last couple of weeks that when I would make a change I wasn’t always getting notified.  I would get some alerts but not others.  So I started with looking at the Nitro appliance to see if it was having a problem.  As I was debugging it I noticed that it wasn’t getting all of the messages to be able to alert off of.  Information is missing and I needed to find it.

I then started looking at my syslog config on my ASA, here it is for reference:

logging enable
logging console informational
logging buffered debugging
logging trap informational
logging host inside x.x.x.x
no logging message 106015
no logging message 313001
no logging message 313008
no logging message 419002
no logging message 106023
no logging message 710003
no logging message 106100
no logging message 302014
no logging message 609002
no logging message 609001
no logging message 302018

Nothing out of the ordinary or so I thought.  As I was using Google to look at stuff I came across some messages about the logging queue limit, which by default is 512.  I decided to look at that and see if that could be causing my issue.  When I looked I saw this:

sh logging queue

Logging Queue length limit : 512 msg(s)
-1123742434 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 512 msg on queue, 512 msgs most on queue

Definitely not good I am dropping messages and the queue is full.  I thought well may be I can increase the queue and everything will be fine after that.  So I did the following:

logging queue 1024
sh logging queue

Logging Queue length limit : 1024 msg(s)
-1123731334 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 1024 msg on queue, 1024 msgs most on queue

Immediately the queue jumped up and I was still dropping messages.

I thought okay may be I need to prune out the message I am logging on.  These are minor messages and I hadn’t needed this data in any of my investigations so I decided to kill it.

no logging message 302017
no logging message 302016
no logging message 302021
no logging message 302020

sh logging queue

Current 1024 msg on queue, 1024 msgs most on queue

No change and I am still dropping messages.  I then thought heck, may be I just need a bigger queue, that always solves the problem right.  Bigger is better:

logging queue 4196
sh logging queue

Logging Queue length limit : 4196 msg(s)
-1123742651 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 4196 msg on queue, 4196 msgs most on queue

That didn’t help out so much.  Immediately I am at a full queue and still dropping messages.  Then I looked back through my config again:

sh run logging

logging enable
logging console informational
logging buffered debugging
logging trap informational
logging queue 4196
logging host inside  x.x.x.x
no logging message 106015
no logging message 313001
no logging message 313008
no logging message 419002
no logging message 106023
no logging message 710003
no logging message 106100
no logging message 302014
no logging message 609002
no logging message 609001
no logging message 302018
no logging message 302017
no logging message 302016
no logging message 302021
no logging message 302020

Hmm, I wonder if logging to the console and buffer are causing my issue.  I am not using them currently and the last time I was troubleshooting I did turn them on.  Could I really have not cleaned up after myself and could this be causing me an issue?  I then did the following:

no logging console
no logging buffered
sh logging queue

Logging Queue length limit : 4196 msg(s)
-1123728024 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 0 msg on queue, 4196 msgs most on queue

Immediately the queue dropped down and there was nothing in it.  I then moved the queue back down to a smaller number.

logging queue 1024
sh logging queue

Logging Queue length limit : 1024 msg(s)
-1123728024 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 0 msg on queue, 4196 msgs most on queue

No messages on queue and no dropped messages.  Also all of my test alerts are now working correctly and everything seems to be fine.

Lesson relearned, when you turn on something make sure you turn it off.  Even if at the time you don’t think it will cause you an issue it may come and cause you an issue later.

Using Nitro to monitor Domain Admin Password Changes

Joseph JenkinsLeave a comment

We had a requirement to monitor when Domain Admins changed their password so that we could pass our PCI Audit for this year. Mostly the issue came out of the fact that a hacker could just as easily change a domain admin password as they could create a new account in domain admins. I was already monitoring domain admins for new accounts getting created, this just seemed like the next step.

I came up with this method to be able to monitor with Nitro and then send me a report.

1. Create a Watchlist
call it whatever you want, but make it a destination user type and then choose an assignee.

2. Input all of your Domain Admins into the Watchlist

3. After creating the watchlist, then go into Alarms and create a new alarm.
Give it a name
choose an assignee

4. Go to the Condition tab and choose “specified event rate”, set the event count to “1”, and choose the time frame as 10 minutes. This is needed to trigger the alarm when you need it to fire.

5. then click on the filter icon, for your query filter choose the following:
signature id: 43-211006270,43-211006280,43-263047230
destination user: WL:(name of watch lists)
Under select a device, choose all of your domain controllers. We have two domains so we are monitoring several machines.

6. Now here is where the magic comes in and the best alerting you can setup. Under actions, choose Generate Report and then click configure
Here you are going to create a new report that lists the userid field and any other information you want. You will also choose your email recipients here so that you can be notified of when a domain admins password has changed. Since the parameters from the alarm will not be passed to the report you will need to choose a few things for the filter at the bottom to keep it to what you are looking for.
signature id: 43-211006270,43-211006280,43-263047230
destination user: WL:(name of watch lists)
set the time range to the last 10 minutes

7. click Finish, you now have an alarm that will be triggered when a domain admin changes their password.

Nitro…err Mcafee Enterprise Security Manager

Joseph JenkinsLeave a comment

We purchased the Nitro Logging Appliance and fell in love with some of it’s features.  Right as we purchased it though Mcafee came in and purchased the company because it was a great product. While Mcafee is all over the product now it doesn’t seem to be lacking in features or functionality.  I have found quite a few things that don’t really seem to be documented anywhere else that I will start putting into my blog.  The first of which on this post will be common commands to run to find out what is going on with the appliance and to make sure that it is working.

Check that logs are coming in:
single host:
tcpdump -nnXi eth0 host (ip of host) and port (syslog port) -s0

Subnet:
tcpdump -nnxi eth0 src net (subnet)/24 and port (syslog port) -s0

The logs won’t be human readable but at least you can see that data is coming in.

Stop and start the Services(this is case sensitive):
NitroStop –nod
NitroStart –nod

I will continue to put up posts and log important stuff here as time goes on.

Upgrading Tippingpoint SMS to 3.5

Joseph Jenkins4 Comments

Today I upgraded our SMS server for Tippingpoint to version 3.5. While this should have been a straight forward upgrade, it nevertheless wasn’t. First of all I had to upgrade to patch 7 of the 3.2 release so that I could supposedly easily make the upgrade to 3.5. I did that upgrade and didn’t have any problems with it However when I tried to use the automated upgrade within the SMS to 3.5 it kept downgrading itself to 3.2 without the patches. I looked through the release notes and couldn’t find anything about this except that there was no upgrade to 3.5 form anything below 3.3, so I tried a few more times to do the update with no success. Finally I just gave up and downloaded the 933Mb full install package from the TMC.tippingpoint.com website and ran the install from that. The install took about 10 minutes.

After letting the full install package run I then logged into the website for the SMS and was able to download the 32bit windows client, I was also pleasantly surprised to find that there is now a Mac OS X client. I loaded the mac client up and was able to login to my SMS with no problems. While I think the interface overall could use some usability enhancements, I was at least happy to find that it kept parity with the Windows version as far as features and functionality. This is now one less reason I need to keep a Windows box around if I can get a VIC client for vmware when 5.1 comes out then I will be extremely happy.