Networking

Going from OC3 to Ethernet

Joseph JenkinsLeave a comment

Currently we have several OC3s that provide access to the internet. One of these OC3s is just too darn expensive so I have decided to move it to a 200Mb ethernet. This is the first Ethernet product that I am installing, mostly I have dealt with Serial circuits such as OC3s and OC12s. This has definitely been a new experience and an interesting one for me. Learning about how they bring the circuit in and getting to see the fiber build out for my area has been a worthwhile experience. Provided my tenure with this company is long having that knowledge will be invaluable for the future projects that I embark on.

Since I am saving money there is more than enough left over so that I can install a backup burst able ds3 with a 1Mb cir. I am pairing this with the provider over bgp to provide redundancy for my dmvpn stores. I also had the ds3 back hauled to San Francisco so that if I have a regional disaster our stores and other distribution center are not completely out of luck.

Both installs went remarkably well and we have achieved both redundancy and cost savings for only a little bit of my engineering time.

Goal Given Up

Joseph JenkinsLeave a comment

If you have read my profile or looked at anything I do then you know that I was working on getting my CCIE in Security. This was a goal that I had started about 5 years ago. I already had my CCSP and thought that this was a good next step for me, I was working at a job that had plenty of equipment for me to work with and would provide me the time to study. I passed the written and had my lab scheduled for a year after that. I purchased the INE training workbooks and video on demand and thought I am ready to start this endeavor. Unfortunately due to job changes and such I put the CCIE on hold, thinking I may never do it again.

2010, Another job change and a chance to pursue this goal. I decided it was important for me to get my CCIE so I started on my trek for it. I still had my old training materials and figured now was the time. I discussed it with my employer and they didn’t want to help me finance this endeavor or provide me with time off to pursue it. So this was going to be self funded goal. I looked at the household finances and thought okay we can afford this. After discussing with my family I got their support and decided to make a go of it. I started down the path again of studying for the written. I put in a good year of study and reading before I took the written again at Cisco Live in 2011. I passed with flying colors and then scheduled my first lab attempt. I scheduled my first attempt for May 1, 2012. That gave me about 10 months of study time it also allowed me to take a class at CCBOOTCAMP about 3 months before the exam. I got home from Cisco Live and immediately started putting together my lab. While I was acquiring the pieces for my lab I started watching all of the INE videos that I had purchased a few years ago. They had updated the videos so I bought the new ones and watched both the old and new multiple times. I also thought that getting some audio content would be good so I ended up purchasing the Audio from IPEXPERT. I spent a good month watching videos and listening to the audio content when I couldn’t watch the vides. I also put together my lab for the INE workbooks, 10 routers, two ASAs, 1 IPS, 2 switches, and one ESX 4.x box that would host both PC and ACS server, not cheap but it would allow me to study whenever I wanted to. I spent the next 2 months doing all of workbook 1 going over each lab until I knew it. Unfortunately since work wasn’t helping me my study times were early in the morning until work and then late at night after work after putting my son to bed. I spent the next several months doing the workbook 2.

January came and it was time for my class at CCBOOTCAMP. I took a week off of work, and I went to LV. I was really afraid that I still wasn’t ready for the test or the class, this class was a chance for me to see how good I was and if I was even close to being ready to take the test on May 1. I still had time to cancel the test and reschedule if needed. The class was great, I learned a couple of things I didn’t know, but for the most part I came away feeling really confident on my skill set and that I would be ready to take the test in 3 months. I got home and while still working on Workbook 2 labs, I decided to intersperse it with Workbook 1 labs and also rewatching all of the videos and listening to the audio content again.

May 1, 2012, I flew up the day before to San Jose. I got to the lab about 20 minutes early and walked around the campus to help calm my nerves. 8:15am the lab starts I open the book and look at the topology and start getting really nervous. I had an issue focusing and not being able to follow the traffic flow. I spent at least a good 30 minutes looking over the topology and calming myself down. When that didn’t work I decided to just start configuring and go from there. At least get some points on the board. That helped immensely I calmed myself down and was able to start working the topology. I spent the next 8 hours working my butt off. As I figured there was at least 1 thing I hadn’t seen before, I found it in the docs and configured it what I thought was correctly. When I left the lab that night I was really confident in what I had done. I thought I had done really well and if I hadn’t passed I had to have been really close.

May 2, 2012, I get my score report and I did horribly. I spent the next day at work wondering how it is that I screwed up so bad, I couldn’t see it. I thought I had done it all correctly, or at least deserved better than I got. I was depressed and felt like a total and utter failure. I was so embarrassed by what I had done that I was ready to give up there. My wife said I should try again, and I said “no”. I said “I have no idea what I missed and it would just be a waste of money”. I spent the next several days in a really depressed funk and not happy at all.

About a week after the test I finally decided to write down everything that was on the test and all of the questions from memory. I started to realize that I had misread stuff and didn’t give them exactly what they wanted. That I had made mistakes and it was my own fault for not passing. I discussed it again with my family and decided that it was worth it to give it another shot. I realized where I had made my mistakes and I thought that I could fix them. I scheduled my next attempt for 2 months out for the beginning of July. We would be on vacation and up in Santa Cruz. I could save myself the travel costs and drive over for the test. I spent the next 2 months solely working on labs, trying for perfection I also threw in Yusuf’s labs from his book, since I had memorized the labs from INE. I also went back over and did the labs from CCBOOTCAMP. I tried to put everything in front of me that could mess me up and give me as much variety as possible.

2nd Attempt, I arrive at the facility and sit down and look over the topology. I am much more comfortable this time. I spend the next 8 hours pounding through the lab, having few difficulties. I leave and believe that everything is working correctly. I get my score report that night this time, while I did much better. I still missed it by a few points. I thought may be something was misgraded and figured what was worth $250, so I asked for the regrade. About 2 weeks later I found out that the score was the same I will not be a CCIE.

As I sit here and write this I have realized that I am not nearly as upset after this attempt as I was my first. Based on factors outside of my control, I am giving up this goal. I don’t have the money to try for a third time, and the reality is even if I wanted to there are no lab dates to be had since v4 was released. I have also enjoyed having my freetime back and spending time with my family again and not always feeling guilty that something is taking away time from studying. I did think about v4 and looked over the topics, based on what is the on the lab and the fact that I will never see most of it in production I am passing at this point. I have sold off my existing lab equipment to pay off my Credit Card that has seen a lot of work through this endeavor. I am not bitter or feel that it was a waste. While I spent a lot of money and time on this and while I didn’t achieve my goal I did learn a lot and did reinforce a heck of a lot of information into my head.

As I look to my next goals, my CCSP expires in November so I need to get cracking if I hope to get my CCNP Security before then.

Brocade Authorization against Cisco ACS 5.3

Joseph Jenkins1 Comment

As I have continued my on the job training with Brocade I came across the fact that we were just using a simple username/password for login.  I decided to integrate it into our production Cisco ACS environment.  Unfortunately there wasn’t a document on doing this for version 5.3.  I had to amend the 4.x information from Brocade.  I figured I would document it here for anyone else that was interested.

I am making the assumption you are:

  1. comfortable with the CLI on Brocade
  2. that you know how to add RADIUS VSA attributes to the ACS server

I used two documents from Brocade to set this up.  The first was Chapter 5 of the FOS admin guide.  This was useful in setting up the Radius Server and setting up the auth parameters.  Namely Radius First, Local second only if the Radius server is not reachable.  Here are the important parts out of the FOS admin guide:

Adding a RADIUS or LDAP server to the switch configuration
1. Connect to the switch and log in using an account with admin permissions.

2. Enter the aaaConfig –add command.

Configuring local authentication as backup
It is useful to enable local authentication so that the switch can take over authentication locally if the RADIUS or LDAP servers fail to respond because of power outage or network problems. Example of enabling local authentication, enter the following command for RADIUS

switch:admin> aaaconfig –authspec “radius;local” –backup

The other document that I used was an ACS_FOS Doc from Brocade.  The important part out of that were the FOS attributes for Radius, so from this:

[User Defined Vendor]
Name=Brocade
IETF Code=1588

VSA 1=Brocade-Auth-Role
VSA 2=Brocade-AVPairs1
VSA 3=Brocade-AVPairs2
VSA 4=Brocade-AVPairs3
VSA 5=Brocade-AVPairs4
VSA 6=Brocade-Passwd-ExpiryDate
VSA 7=Brocade-Passwd-WarnPeriod

[Brocade-Auth-Role]
Type=STRING
Profile=OUT
[Brocade-AVPairs1]
Type=STRING
Profile=OUT
[Brocade-AVPairs2]
Type=STRING
Profile=OUT
[Brocade-AVPairs3]
Type=STRING
Profile=OUT
[Brocade-AVPairs4]
Type=STRING
Profile=OUT
[Brocade-Passwd-ExpiryDate]
Type=STRING
Profile=OUT
[Brocade-Passwd-WarnPeriod]
Type=INTEGER
Profile=OUT

I set up this:

The other part that was useful was this section on what to put in for the admin section:

Under “Brocade-Auth-Role” enter “admin” for administrator

This value is the pre-defined user roles in FOS. Examples are:

admin
basicswitchadmin
user
switchadmin
zoneadmin
operator
fabricadmin
securityadmin

Which on ACS gave me this:

Now based on the User that logins I can control what access they have into our Brocade environment. This also has the added benefit of allowing us to audit the users that are logging into Brocade so that I don’t have to worry about changes being made by unauthorized/authorized people. I hope this helps someone else and saves them from having to hunt around and find the information.

Troubleshooting VEEAM and MS Exchange running on VMware ESX

Joseph JenkinsLeave a comment

While troubleshooting some issues with Veeam running backups of our MS Exchange environment came across an issue where it was failing with the following error:

Freezing guest operating system
Unfreeze error (over VIX): [Backup job failed.]

The only info MS Exchange itself was giving was that VSS was timing out. What I found out that we had to do was determine what VSS writers were failing. That is when I came across this command:

vssadmin list writers

At a normal time all of the writes should show up as stable and ok. However our Microsoft Exchange Writer was showing as “Timed Out”

What I ended up doing was stopping and starting the information store which resolved the VSS error and then reran the command “vssadmin list writers” and saw that everything was okay at that point. I was then able to rerun the Veeam backup and get a good backup of our Exchange environment. Please note that stopping and starting the information store will cause your databases to become unavailable for a time.

Cryptocard and Anyconnect are my bain at 2am

Joseph JenkinsLeave a comment

we use Cryptocard as our security authentication mechanism at my company.  That what you have and the what you know.  I recently ran into an issue with the hardware tokens that was quite perplexing and caused me some grief at 2am.  We require a vpn from the inside of our company to get to PCI systems, this ensures that no one is listening on the wire and isn’t trying to steal our credit card or other vital data.  So my night operators need to use VPN along with Cryptocard to login to just about everything they want to do.  So this morning at 2am I got a call from one of the night operators who was trying to login to the VPN and getting a prompt to log back in again.  When logging into the Cryptocard status screen I saw this for his status:

Unfortunately it doesn’t seem like the Cisco Anyconnect client doesn’t seem to be able to handle the next token code appropriately and the user was getting a login failed message.

Usually when we have this problem with the software tokens we just resync the users token with the console.  No big deal except trying to get a user to enter a 8 digit challenge code that consists of upper/lower case and special characters can be tough.

However we are now switching to the hardware tokens and they only have one button on them.  This makes it extremely tough to enter a challenge code into the hardware token.  In this case since it was an internal user I just gave them the lowest possible operator rights “snapshot” in the console and allowed them to login there.  Then when it asked for the next code they were able to put it in and get their token code + pin and it resynced correctly.  Not sure what I will do when a user is remote with a hardware token and can’t login.  Probably have to issue them a software token at that point and then work with them when they are back in the office.

Upgrade Cisco ACS from 5.2 to 5.3

Joseph Jenkins3 Comments

I am in the process of upgrading my ACS deployment from 5.2 to 5.3.  When I first got to this company we had one 3.x ACS Appliance that was woefully unsupported and out of date.  Since I started I managed to purchase two vmware servers with the large deployment license.  So my configuration is this:

Primary Server + Log Collector at the main site

Secondary Server at our remote site

As I read through the documentation for the upgrade and tried to understand Cisco’s convoluted process of actually upgrading stuff, I came to a stark realization.  The DB from 5.2 can be directly imported into 5.3….

I quickly ditched all plans I had to follow Cisco’s cruddy upgrade process and just made sure all of my equipment had both the primary and secondary ACS servers setup.  I then created a brand new 5.3 server and performed all of the necessary patches to get it to the latest and greatest.  After that I exported my 5.2 database then imported it into 5.3.  Once I had done some testing and was sure my new 5.3 was good I then shutdown the 5.2 Primary and swapped the ip to the new Primary 5.3.  I then did the same process for my secondary and then restarted the distributed database.

Much easier upgrade and didn’t require more than 5 minutes of downtime and since I had two ACS’s nothing was missed during the time and no one was denied access to the network.

Thanks Cisco for making at least one thing in life easy and keeping the databases compatible.

When HSRP tracking an interface isn’t enough

Joseph JenkinsLeave a comment

I ran into an issue where I have HSRP going between two of my internet routers that are both using BGP to receive their default route and propagate out the routes for my network. I was tracking the WAN interface so that if it went down on either one side it would then move the HSRP addresses over to the other router. The issue I ran into was that my upstream BGP peer died and the interface stayed up for my primary connection.

So what issue did this cause? This made it so that HSRP addresses didn’t move since the interface didn’t go down.

I decided to dive into the cisco bag of tricks and figure out another way to monitor the BGP peer being up or down. Unfortunately this wasn’t something I could find in the 12.4 code, what I did find was the ability to monitor for a route being available. So since I only take a default route from the upstream BGP neighbors and my two connections are homed back to separate cities and BGP peers. I put in the following:

track 101 ip route 0.0.0.0 0.0.0.0 reachability

This allows the router to track for the default route being in the table, which if the BGP peer goes away so will this route. This can then shift the traffic over to the other internet connection and allow my access to the internet to live.

A simple sh track, shows that everything is working as it supposed to:

Track 101
IP route 0.0.0.0 0.0.0.0 reachability
Reachability is Up (BGP)
2 changes, last change 5d04h
First-hop interface is ATM4/0.148
Tracked by:
HSRP FastEthernet0/1 1
HSRP FastEthernet0/1 2
HSRP FastEthernet0/1 3
HSRP FastEthernet0/1 4
HSRP FastEthernet0/1 5
HSRP FastEthernet0/1 6

Simple clean and effective.

And now for something completely different – Storage with Brocade

Joseph JenkinsLeave a comment

So as attrition has struck and we have lost staff, I am now taking over Storage duties.  It’s a switch right, it should work just like a network switch.  That is the thinking of people above me, I think they have been away from hands on work for too long. I understand iSCSI, that’s simple.  This FC stuff, I have a bit of a curve in front of me.

First order of the day….  Assess where we are and if we need a firmware update to maintain our pci compliance.  Nothing like getting thrown in and going from there.  Well at least I know we have Brocade Swithces so I am starting off with at least one advantage.   So after finding the username and password to log into them, I then needed to find out what version we had.  This website was invaluable in determing that:

http://www.boredsysadmin.com/2009/05/how-to-find-out-brocade-switch-model.html

If you scroll down to the comments someone has updated with the later versions, but I put the instructions here just as a quick look along with the meanings of the different Switchtypes.

Open web browser at http://hostname of switch/SwitchInfo.html

While I am surprised about the amount of information that is given away for free, I am glad that it is at least available for me to help me out.

Scroll down till you see in List of Ports: switchType:    xx.x

Switch Type Switch Name translation for xx.x
1 Brocade 1000 Switches
2,6 Brocade 2800 Switch
3 Brocade 2100, 2400 Switches
4 Brocade 20×0, 2010, 2040, 2050 Switches
5 Brocade 22×0, 2210, 2240, 2250 Switches
Switch Types and Product Names
Generated by Jive SBS on 2011-01-06-07:00
2
7 Brocade 2000 Switch
9 Brocade 3800 Switch
10 Brocade 12000 Director
12 Brocade 3900 Switch
16 Brocade 3200 Switch
17 Brocade 3800VL
18 Brocade 3000 Switch
21 Brocade 24000 Director
22 Brocade 3016 embedded Blade Switch
23 8Gbit 10-port embedded fabric switch
26 Brocade 3850 Switch
27 Brocade 3250 Switch
29 Brocade 4012 Embedded Blade Switch
32 Brocade 4100 Switch
33 Brocade 3014 Switch
34 Brocade 200E Switch
36 Brocade FR4-18i Director Blade
37 Brocade 4020 Embedded Blade Switch
38 Brocade 7420 SAN Router
40 Fibre Channel Routing (FCR) Front Domain
41 Fibre Channel Routing (FCR) Xlate Domain
42 Brocade 48000 Director
43 Brocade 4024 Embedded Blade Switch
44 Brocade 4900 Switch
45 Brocade 4016 Embedded Blade Switch
46 Brocade 7500 Switch
Switch Types and Product Names
Generated by Jive SBS on 2011-01-06-07:00
3
47 Brocade FC4-16IP Director Blade
50 Brocade 4GB FC Port Blade
51 Brocade 4018 Embedded Blade Switch
55 Brocade FA4-18i Extension Director Blade
55,2 Brocade 7600 Switch
58 Brocade 5000 Switch
62 Brocade DCX Backbone
63 Brocade 8Gb Backbone Core Fabric Switch
64 Brocade 5300 Switch
66 Brocade 5100 Switch
67 Brocade Encryption Switch
68 Brocade 8Gb 16 FC 2 GigE ports Director Encryption
Blade
69 Brocade 5410 Blade
70 Brocade 8GB 10 Port Embedded Fabric Switch
71 Brocade 300 Switch
72 Brocade 5480 Embedded Blade Switch
75 Brocade M5424 Embedded Blade Switch
76,6 Brocade 8000 FCoE Switch
77,3 Brocade DCX-4S
82 Brocade 8Gb 24-port Embedded Blade Switch
83 Brocade 16-FC port, 6-GE port, auto sensing 1, 2, 4 or
8Gbit Switch
86 Brocade 8Gbit 26-port embedded Switch
88 Brocade 10Gb 24 GigE ports DCE Blade
Switch Types and Product Names
Generated by Jive SBS on 2011-01-06-07:00
4
89 Brocade 8Gb 12 FC, 1Gb 10 GigE FCIP Blade, 10Gb 2
GigE ports FCR
Now at least I am off and running to the Brocade site to find out exactly how many versions I am back.
I also needed to determine the serial number so as I found out that “?” didn’t do anything for me, I then tried “help” and that listed all of the commands on the switch quite a few.  I figured the command had to start with switch, chassis, or hardware.  I looked through and was able to find this command:
“chassisshow”
Luckily this gives me everything I need now and can open my support case or get access to the support site to find the information for myself.

Zterm has been updated….

Joseph JenkinsLeave a comment

I have been using zterm on my mac for years and was upset to find that when I upgraded to Lion it wasn’t going to come with me. It is a simple no nonsense console utility that works well with my Keyspan USB to Serial adapter. After looking around though the author has updated it to be a universal binary which can be had here:

http://homepage.mac.com/dalverson/zterm/

Great application for a great platform. If you use it and you like it, flip the author some coffee money and let him know how much you like it.

Studying for CCIE Security and using GNS3 for now

Joseph JenkinsLeave a comment

I am in the process of studying for my CCIE Security Lab and at this point I am still trying to collect hardware as cheaply as possible, since money is an object for me.  So in the meantime I am working with GNS3 and trying to use virtual equipment to help me along and get some of my studies accomplished.  So far I have ran into a few issues that are killing me:

1. The issue with c3700 units.  No matter what I did I couldn’t save the configurations out of the 3700s.  Turns out there is a bug in the code that doesn’t allow you to save the configurations to the startup config for this model of router.  Solution, don’t use them.

2. Transparent mode in the ASAs doesn’t work at all.  You can create the configuration for the ASA and put it into transparent mode, good luck getting it to pass traffic.  Solution, buy em or rent em.

3. The setup for Micro Linux running in QEMU.  Got it loaded and hooked up to a router, but actually configuring the interface was a little more of a pain.  Logging in as root isn’t the same thing as logging in as root on a normal linux machine.  Here is brief snippet for configuring a Micro Linux instance and giving it an IP address.

tc@box:~$ sudo su
root@box:~# ifconfig eth0 10.0.0.100 netmask 255.255.255.0 up
root@box:~# route add default gw 10.0.0.1

4. Multiple context mode is a no go on the ASA as well.  Solution, buy em or rent em.  Looking for some cheap prices on them at this point.

Other than these issues things are working out pretty well on my virtual lab.  I still have a long road ahead of me before I will be ready to take the test.  I am working on speed at this point and doing the workbooks from INE.  I hope to be ready by Jan or Feb of next year.

Mostly put this together so that anyone else trying to do what I am doing can find the information in one place.