Going from OC3 to Ethernet

Currently we have several OC3s that provide access to the internet. One of these OC3s is just too darn expensive so I have decided to move it to a 200Mb ethernet. This is the first Ethernet product that I am installing, mostly I have dealt with Serial circuits such as OC3s and OC12s. This has definitely been a new experience and an interesting one for me. Learning about how they bring the circuit in and getting to see the fiber build out for my area has been a worthwhile experience. Provided my tenure with this company is long having that knowledge will be invaluable for the future projects that I embark on.

Since I am saving money there is more than enough left over so that I can install a backup burst able ds3 with a 1Mb cir. I am pairing this with the provider over bgp to provide redundancy for my dmvpn stores. I also had the ds3 back hauled to San Francisco so that if I have a regional disaster our stores and other distribution center are not completely out of luck.

Both installs went remarkably well and we have achieved both redundancy and cost savings for only a little bit of my engineering time.

Goal Given Up

If you have read my profile or looked at anything I do then you know that I was working on getting my CCIE in Security. This was a goal that I had started about 5 years ago. I already had my CCSP and thought that this was a good next step for me, I was working at a job that had plenty of equipment for me to work with and would provide me the time to study. I passed the written and had my lab scheduled for a year after that. I purchased the INE training workbooks and video on demand and thought I am ready to start this endeavor. Unfortunately due to job changes and such I put the CCIE on hold, thinking I may never do it again.

2010, Another job change and a chance to pursue this goal. I decided it was important for me to get my CCIE so I started on my trek for it. I still had my old training materials and figured now was the time. I discussed it with my employer and they didn’t want to help me finance this endeavor or provide me with time off to pursue it. So this was going to be self funded goal. I looked at the household finances and thought okay we can afford this. After discussing with my family I got their support and decided to make a go of it. I started down the path again of studying for the written. I put in a good year of study and reading before I took the written again at Cisco Live in 2011. I passed with flying colors and then scheduled my first lab attempt. I scheduled my first attempt for May 1, 2012. That gave me about 10 months of study time it also allowed me to take a class at CCBOOTCAMP about 3 months before the exam. I got home from Cisco Live and immediately started putting together my lab. While I was acquiring the pieces for my lab I started watching all of the INE videos that I had purchased a few years ago. They had updated the videos so I bought the new ones and watched both the old and new multiple times. I also thought that getting some audio content would be good so I ended up purchasing the Audio from IPEXPERT. I spent a good month watching videos and listening to the audio content when I couldn’t watch the vides. I also put together my lab for the INE workbooks, 10 routers, two ASAs, 1 IPS, 2 switches, and one ESX 4.x box that would host both PC and ACS server, not cheap but it would allow me to study whenever I wanted to. I spent the next 2 months doing all of workbook 1 going over each lab until I knew it. Unfortunately since work wasn’t helping me my study times were early in the morning until work and then late at night after work after putting my son to bed. I spent the next several months doing the workbook 2.

January came and it was time for my class at CCBOOTCAMP. I took a week off of work, and I went to LV. I was really afraid that I still wasn’t ready for the test or the class, this class was a chance for me to see how good I was and if I was even close to being ready to take the test on May 1. I still had time to cancel the test and reschedule if needed. The class was great, I learned a couple of things I didn’t know, but for the most part I came away feeling really confident on my skill set and that I would be ready to take the test in 3 months. I got home and while still working on Workbook 2 labs, I decided to intersperse it with Workbook 1 labs and also rewatching all of the videos and listening to the audio content again.

May 1, 2012, I flew up the day before to San Jose. I got to the lab about 20 minutes early and walked around the campus to help calm my nerves. 8:15am the lab starts I open the book and look at the topology and start getting really nervous. I had an issue focusing and not being able to follow the traffic flow. I spent at least a good 30 minutes looking over the topology and calming myself down. When that didn’t work I decided to just start configuring and go from there. At least get some points on the board. That helped immensely I calmed myself down and was able to start working the topology. I spent the next 8 hours working my butt off. As I figured there was at least 1 thing I hadn’t seen before, I found it in the docs and configured it what I thought was correctly. When I left the lab that night I was really confident in what I had done. I thought I had done really well and if I hadn’t passed I had to have been really close.

May 2, 2012, I get my score report and I did horribly. I spent the next day at work wondering how it is that I screwed up so bad, I couldn’t see it. I thought I had done it all correctly, or at least deserved better than I got. I was depressed and felt like a total and utter failure. I was so embarrassed by what I had done that I was ready to give up there. My wife said I should try again, and I said “no”. I said “I have no idea what I missed and it would just be a waste of money”. I spent the next several days in a really depressed funk and not happy at all.

About a week after the test I finally decided to write down everything that was on the test and all of the questions from memory. I started to realize that I had misread stuff and didn’t give them exactly what they wanted. That I had made mistakes and it was my own fault for not passing. I discussed it again with my family and decided that it was worth it to give it another shot. I realized where I had made my mistakes and I thought that I could fix them. I scheduled my next attempt for 2 months out for the beginning of July. We would be on vacation and up in Santa Cruz. I could save myself the travel costs and drive over for the test. I spent the next 2 months solely working on labs, trying for perfection I also threw in Yusuf’s labs from his book, since I had memorized the labs from INE. I also went back over and did the labs from CCBOOTCAMP. I tried to put everything in front of me that could mess me up and give me as much variety as possible.

2nd Attempt, I arrive at the facility and sit down and look over the topology. I am much more comfortable this time. I spend the next 8 hours pounding through the lab, having few difficulties. I leave and believe that everything is working correctly. I get my score report that night this time, while I did much better. I still missed it by a few points. I thought may be something was misgraded and figured what was worth $250, so I asked for the regrade. About 2 weeks later I found out that the score was the same I will not be a CCIE.

As I sit here and write this I have realized that I am not nearly as upset after this attempt as I was my first. Based on factors outside of my control, I am giving up this goal. I don’t have the money to try for a third time, and the reality is even if I wanted to there are no lab dates to be had since v4 was released. I have also enjoyed having my freetime back and spending time with my family again and not always feeling guilty that something is taking away time from studying. I did think about v4 and looked over the topics, based on what is the on the lab and the fact that I will never see most of it in production I am passing at this point. I have sold off my existing lab equipment to pay off my Credit Card that has seen a lot of work through this endeavor. I am not bitter or feel that it was a waste. While I spent a lot of money and time on this and while I didn’t achieve my goal I did learn a lot and did reinforce a heck of a lot of information into my head.

As I look to my next goals, my CCSP expires in November so I need to get cracking if I hope to get my CCNP Security before then.

Brocade Authorization against Cisco ACS 5.3

As I have continued my on the job training with Brocade I came across the fact that we were just using a simple username/password for login.  I decided to integrate it into our production Cisco ACS environment.  Unfortunately there wasn’t a document on doing this for version 5.3.  I had to amend the 4.x information from Brocade.  I figured I would document it here for anyone else that was interested.

I am making the assumption you are:

  1. comfortable with the CLI on Brocade
  2. that you know how to add RADIUS VSA attributes to the ACS server

I used two documents from Brocade to set this up.  The first was Chapter 5 of the FOS admin guide.  This was useful in setting up the Radius Server and setting up the auth parameters.  Namely Radius First, Local second only if the Radius server is not reachable.  Here are the important parts out of the FOS admin guide:

Adding a RADIUS or LDAP server to the switch configuration
1. Connect to the switch and log in using an account with admin permissions.

2. Enter the aaaConfig –add command.

Configuring local authentication as backup
It is useful to enable local authentication so that the switch can take over authentication locally if the RADIUS or LDAP servers fail to respond because of power outage or network problems. Example of enabling local authentication, enter the following command for RADIUS

switch:admin> aaaconfig –authspec “radius;local” –backup

The other document that I used was an ACS_FOS Doc from Brocade.  The important part out of that were the FOS attributes for Radius, so from this:

[User Defined Vendor]
Name=Brocade
IETF Code=1588

VSA 1=Brocade-Auth-Role
VSA 2=Brocade-AVPairs1
VSA 3=Brocade-AVPairs2
VSA 4=Brocade-AVPairs3
VSA 5=Brocade-AVPairs4
VSA 6=Brocade-Passwd-ExpiryDate
VSA 7=Brocade-Passwd-WarnPeriod

[Brocade-Auth-Role]
Type=STRING
Profile=OUT
[Brocade-AVPairs1]
Type=STRING
Profile=OUT
[Brocade-AVPairs2]
Type=STRING
Profile=OUT
[Brocade-AVPairs3]
Type=STRING
Profile=OUT
[Brocade-AVPairs4]
Type=STRING
Profile=OUT
[Brocade-Passwd-ExpiryDate]
Type=STRING
Profile=OUT
[Brocade-Passwd-WarnPeriod]
Type=INTEGER
Profile=OUT

I set up this:

The other part that was useful was this section on what to put in for the admin section:

Under “Brocade-Auth-Role” enter “admin” for administrator

This value is the pre-defined user roles in FOS. Examples are:

admin
basicswitchadmin
user
switchadmin
zoneadmin
operator
fabricadmin
securityadmin

Which on ACS gave me this:

Now based on the User that logins I can control what access they have into our Brocade environment. This also has the added benefit of allowing us to audit the users that are logging into Brocade so that I don’t have to worry about changes being made by unauthorized/authorized people. I hope this helps someone else and saves them from having to hunt around and find the information.

Troubleshooting VEEAM and MS Exchange running on VMware ESX

While troubleshooting some issues with Veeam running backups of our MS Exchange environment came across an issue where it was failing with the following error:

Freezing guest operating system
Unfreeze error (over VIX): [Backup job failed.]

The only info MS Exchange itself was giving was that VSS was timing out. What I found out that we had to do was determine what VSS writers were failing. That is when I came across this command:

vssadmin list writers

At a normal time all of the writes should show up as stable and ok. However our Microsoft Exchange Writer was showing as “Timed Out”

What I ended up doing was stopping and starting the information store which resolved the VSS error and then reran the command “vssadmin list writers” and saw that everything was okay at that point. I was then able to rerun the Veeam backup and get a good backup of our Exchange environment. Please note that stopping and starting the information store will cause your databases to become unavailable for a time.

Cryptocard and Anyconnect are my bain at 2am

we use Cryptocard as our security authentication mechanism at my company.  That what you have and the what you know.  I recently ran into an issue with the hardware tokens that was quite perplexing and caused me some grief at 2am.  We require a vpn from the inside of our company to get to PCI systems, this ensures that no one is listening on the wire and isn’t trying to steal our credit card or other vital data.  So my night operators need to use VPN along with Cryptocard to login to just about everything they want to do.  So this morning at 2am I got a call from one of the night operators who was trying to login to the VPN and getting a prompt to log back in again.  When logging into the Cryptocard status screen I saw this for his status:

Unfortunately it doesn’t seem like the Cisco Anyconnect client doesn’t seem to be able to handle the next token code appropriately and the user was getting a login failed message.

Usually when we have this problem with the software tokens we just resync the users token with the console.  No big deal except trying to get a user to enter a 8 digit challenge code that consists of upper/lower case and special characters can be tough.

However we are now switching to the hardware tokens and they only have one button on them.  This makes it extremely tough to enter a challenge code into the hardware token.  In this case since it was an internal user I just gave them the lowest possible operator rights “snapshot” in the console and allowed them to login there.  Then when it asked for the next code they were able to put it in and get their token code + pin and it resynced correctly.  Not sure what I will do when a user is remote with a hardware token and can’t login.  Probably have to issue them a software token at that point and then work with them when they are back in the office.

Random Links for the week of 6/2/2012

This is just cool, I have always been a fan of Batman. What geek isn’t considering all of the cool toys:

Attack of the Batmobiles! [Video]

No kidding:

http://www.geeksaresexy.net/2012/04/05/finding-old-friends-with-facebook-comic/

This is very cool:

http://code.google.com/p/ostinato/

This would have been useful information before the lunar eclipse that happened a few weeks ago:

http://www.nikonusa.com/Learn-And-Explore/Photography-Techniques/h1sctsrv/1/How-to-Photograph-a-Lunar-Eclipse.html?cid=eml-0612-lenewsletter-v1-featuredlm

Upgrade Cisco ACS from 5.2 to 5.3

I am in the process of upgrading my ACS deployment from 5.2 to 5.3.  When I first got to this company we had one 3.x ACS Appliance that was woefully unsupported and out of date.  Since I started I managed to purchase two vmware servers with the large deployment license.  So my configuration is this:

Primary Server + Log Collector at the main site

Secondary Server at our remote site

As I read through the documentation for the upgrade and tried to understand Cisco’s convoluted process of actually upgrading stuff, I came to a stark realization.  The DB from 5.2 can be directly imported into 5.3….

I quickly ditched all plans I had to follow Cisco’s cruddy upgrade process and just made sure all of my equipment had both the primary and secondary ACS servers setup.  I then created a brand new 5.3 server and performed all of the necessary patches to get it to the latest and greatest.  After that I exported my 5.2 database then imported it into 5.3.  Once I had done some testing and was sure my new 5.3 was good I then shutdown the 5.2 Primary and swapped the ip to the new Primary 5.3.  I then did the same process for my secondary and then restarted the distributed database.

Much easier upgrade and didn’t require more than 5 minutes of downtime and since I had two ACS’s nothing was missed during the time and no one was denied access to the network.

Thanks Cisco for making at least one thing in life easy and keeping the databases compatible.

Random Links for the week:

Sometimes fun, sometimes not:

This is such a cool idea why hasn’t someone thought of it over here:
http://wish.co.uk/zombie-shopping-mall/

These are on my list…. Now:
http://failblog.org/2012/03/28/epic-win-photos-win-life-goals-win/

Nice Tattoo’s

Hot Harry Potter Tattoos [Pic]

Really need to take a vacation:

Disney Fantasy – Staterooms and Suites

Important things:
https://learningnetwork.cisco.com/message/184717#184717

My son loved this site:
http://www.amazingpaperairplanes.com/

REGEX – What a pain…

As I have embarked on my path towards the CCIE I have come across a lot that I am not familiar with.  One of those that has given me great pause has been regex.  At its surface it seems simple enough text matching.  As I have gone through my studies it has been anything but.  I am still learning all of the nuances of this and luckily I have had some projects at work that have helped strengthen that knowledge.  So here are some of the regex’s that I have been working on for work and why I did them.

We are whitelisting allowed websites on our Websense appliance initially I went with this webex:

(http://|https://) *.facebook.com

Yes we do allow Facebook from our registers, we are a hip social company and our stores need to be able to inform the kids as to what is going on.

However I ran into an issue when some did something like this and was able to bypass the whitelist or I guess I should say was able to use the whitelist to get out:

http://www.yahoo.com/?www.facebook.com

I then did some looking and realized what I needed to do was disallow special characters before the domain.  DNS doesn’t allow for certain special characters to be in the domain name and they should never show up between http:// or https:// and the domain name, so I then came up with this:

(http://|https://)[^/?//()]*.facebook.com

That managed to fix everything, the pen tester’s weren’t able to get past it and it managed to save us a fair bit of trouble.  However I ran into one additional problem, what happens if the website or the user didn’t put a subdomain in front of facebook.com?  Well then it would fail.  So I ended up going with this regex which has solved all of my problems and everything seems to be working fine now.

(http://|https://)[^/?//()]*facebook.com

Throughout all of this I was using the ASA as my primary vehicle for testing the regex expressions.  With the command:

Test regex (regex) (pattern to match)

caasa1(config)# test regex http://www.facebook.com “(http://|https://)[^///()]*facebook.com”
INFO: Regular expression match succeeded.

caasa1(config)# test regex http://www.yahoo.com/?www.facebook.com “(http://|https://)[^///()?]*facebook.com”
INFO: Regular expression match failed.

The only main gotcha with this is remember to hit ctrl+v before you put in the ? or the ASA will think you are querying for help and take you to the help menu.

When HSRP tracking an interface isn’t enough

I ran into an issue where I have HSRP going between two of my internet routers that are both using BGP to receive their default route and propagate out the routes for my network. I was tracking the WAN interface so that if it went down on either one side it would then move the HSRP addresses over to the other router. The issue I ran into was that my upstream BGP peer died and the interface stayed up for my primary connection.

So what issue did this cause? This made it so that HSRP addresses didn’t move since the interface didn’t go down.

I decided to dive into the cisco bag of tricks and figure out another way to monitor the BGP peer being up or down. Unfortunately this wasn’t something I could find in the 12.4 code, what I did find was the ability to monitor for a route being available. So since I only take a default route from the upstream BGP neighbors and my two connections are homed back to separate cities and BGP peers. I put in the following:

track 101 ip route 0.0.0.0 0.0.0.0 reachability

This allows the router to track for the default route being in the table, which if the BGP peer goes away so will this route. This can then shift the traffic over to the other internet connection and allow my access to the internet to live.

A simple sh track, shows that everything is working as it supposed to:

Track 101
IP route 0.0.0.0 0.0.0.0 reachability
Reachability is Up (BGP)
2 changes, last change 5d04h
First-hop interface is ATM4/0.148
Tracked by:
HSRP FastEthernet0/1 1
HSRP FastEthernet0/1 2
HSRP FastEthernet0/1 3
HSRP FastEthernet0/1 4
HSRP FastEthernet0/1 5
HSRP FastEthernet0/1 6

Simple clean and effective.