Check your Raid: Cisco ASA and Sourcefire

Over the past week I had an issue where one of my Cisco 5545 with a Sourcefire module went down and failed and I couldn’t get it restarted. When I looked at the console for the SFR Module I saw disk errors and I opened a ticket with Cisco to have them look at it. One thing that I found appalling was the quality of Cisco TAC engineers has dropped dramatically. I spent more time on the phone with these guys not knowing what to do and and my showing them commands that I had just googled and what needed to be done. If these guys are supposed to be the experts in the device and technology I am not impressed. Especially since Cisco keeps raising my rates and the quality seems to get lower rather than better.

Back to the issue:

The Cisco 5545 Sourcefire unit has two SSDs in a Raid 1 configuration, so you would think that if one failed the other would take over. At least that is what I thought, however it turns out that both of the SSDs had failed and there was no notification at all on the unit itself or in the logs as to one of the units being bad, let alone both of them. The only place I found it was running the “sh raid” command on the terminal. After seeing the failure of this unit, I then went through the rest of my 5545s with Sourcefire modules and found two others that had a failed drive and there was no warning, no error lights on the drive or the firewall itself. I had to run the command to find the issue.

Here is what a healthy raid set looks like:

Version : 1.2
Creation Time : Fri Feb 19 18:27:16 2021
Raid Level : raid1
Array Size : 124969216 (119.18 GiB 127.97 GB)
Used Dev Size : 124969216 (119.18 GiB 127.97 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Wed Jun  2 20:05:01 2021
      State : clean

Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

   Name : ciscoasa:0  (local to host ciscoasa)
   UUID : 244baa9a:b6e40506:f7384510:fcb42706
 Events : 12123

Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
2 8 16 1 active sync /dev/sdb

Here’s what an unhealthy raid set looks like:

        Version : 1.2
  Creation Time : Mon May 25 12:42:13 2020
     Raid Level : raid1
     Array Size : 124969216 (119.18 GiB 127.97 GB)
  Used Dev Size : 124969216 (119.18 GiB 127.97 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Jun  2 20:01:32 2021
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0

           Name : ciscoasa:0  (local to host ciscoasa)
           UUID : 0ed2ca7c:260897dd:f183f4bf:c0f15bfb
         Events : 12258234

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       2       0        0        2      removed

       2       8       16        -      faulty   /dev/sdb

I can’t believe there is no logs or notifications, Solarwinds didn’t pick up the hardware issues. You would think there would be some sort of notification sent out or the HD light on the firewall would turn orange, what a novel concept to notify people of a failed hardware item before it causes major problems.

So if you run any of the modules in your ASA firewalls, make sure to check the raid level and that the drives are in a healthy state, if not get the ticket open with TAC. Where they can give you such brilliant ideas as move the faulty drive to another ASA, or swap the drives(which causes the firewall to crash, so don’t do it).

Script to update Address entries in Fortigate

With our VPN being over utilized I had to implement split tunneling on our vpn. However there are some web services that require a known IP address to access. Unfortunately these services are on AWS which the IP address changes often. I built this script to lookup the addresses and then update the Fortigate firewalls VPN Routing list to make sure that the traffic goes over the vpn tunnel and through our known IP address to access the service.


#Update Epsilon on the Fortigate Firewall VPNs

from nslookup import Nslookup
from netmiko import ConnectHandler
import cred

device1 = {
    “host”: cred.hostname,
    “username”: cred.rancid_username,
    “password”: cred.rancid_password,
    “device_type”: “fortinet”,
    “secret”: cred.rancid_password,

#Connect to the Fortinet
net_connect = ConnectHandler(**device1)

#Listing of the domains to query
DOMAIN_FILE = open(“domains.txt”, “r”)
#DNS Server to query
DNS_SERVER = [‘x.x.x.x’]

    #queries the specified dns server to get the info for the urls and writes the data to a config file
    dns_query = Nslookup(dns_servers=(DNS_SERVER))
    ips_record = dns_query.dns_lookup(line)
        for x in ips_record.answer:
            FILE_CONFIG.write(“edit ” + line + “_” + str(ORDERNUMBER) + ‘\n’)
            FILE_CONFIG.write(“set subnet ” + x + ”″ +’\n’)
            FILE_CONFIG.write(“next” +’\n’)


FILE_CONFIG = open(‘config.txt’,’w’)
FILE_CONFIG.write(“config firewall address” +’\n’)

for line in DOMAIN_FILE:
line = line.rstrip(‘\n’)

FILE_CONFIG.write(“end” +’\n’)


# write to the Fortigate
output2 = net_connect.send_config_from_file(config_file=”config.txt”)

Microsoft MFA login with Fortigate and Forticlient for SSLVPN

Since I am tired of being a beta tester for Cisco products. I decided to try a different firewall this time around for my company. I looked at both Fortigate and Palo Alto as they seemed to be the leaders in the market right now. I did a bake off for features/functionality vs cost and Fortigate came out as the winner. The firewall was implemented with minimal issues and has been working flawlessly for us. While we were on this project we are also in the process of moving to Azure AD so I decided to combine the Microsoft MFA with our new firewall/vpn solution to save ourselves some money since then we wouldn’t need another 2 factor solution.

I went through the documentation from Fortigate and Microsoft on setting up the SAML authentication and it was pretty good for the most part. Here was the main document that I followed to get everything setup:
I did run into a few issues that I had to fix to get everything working with group memberships, so that users would be enabled to login based on their group and would have the correct policy applied to them.

Here are some things to be aware of and the changes I needed to make:

1. You must be on the 6.4.x code for Fortigate. There are issues with the lower code versions and SAML not working correctly or populating the tables with the necessary information.
2. Wipe out all of the extra entries under Users and Attributes Claims in Azure AD. This is all you should have:
3. Here is the necessary configuration on the Fortigate side:
config user saml
edit “azure”
set cert “Fortinet_Factory”
set entity-id “https://XXXXXXX/remote/saml/metadata”
set single-sign-on-url “https://XXXXXX/remote/saml/login”
set single-logout-url “https://SSSSSSSS/remote/saml/logout”
set idp-entity-id “”
set idp-single-sign-on-url “”
set idp-single-logout-url “”
set idp-cert “REMOTE_Cert_2”
set user-name “username”
set group-name “group”

After these changes everything worked perfectly, I am now in the process of rolling out our new vpn to the users in the company along with the Microsoft MFA client.

How to rebuild an F5 Physical Load Balancer

Because I forget this and it always seems to cause me more pain than it should to have to rebuild one. I’ve had this happen 3 times in the 8 years of dealing with the physical 1600 LTMs all of them have failed due to some power problem that won’t let them startup completely and I end up spending 8 or more hours having to rebuild them and figure out what the heck happened to them. Luckily they have always been in a fault tolerant pair so I haven’t been down completely, but have never wanted to push the amount of time one is down because of how important they are to my company.


Call into Support and open a ticket with the s/n of the failed unit and the error message on the screen.

If you don’t already have enhanced 4 hour replacement ask for an upgrade to it via credit card. Waiting more than 4 hours is very painful and dangerous for us.

Wait 4 hours for the new unit to come in.

While waiting:
Unrack the currently failed unit making sure that all of the cables are correctly labeled and ready to be plugged into the new unit.

Download the current version ISO along with any hot fixes to match the current install version. Download your latest backup for the unit and have it all ready and waiting to go on your laptop.

On the active unit make sure to clear out any ssh keys if needed from the failover interface

Also Reset the Device Trust under Device Management/Device Trust on the active unit

When the new unit finally arrives rack it and plug in at least the serial cable and the management ethernet cable. Before powering on plug in the recovery USB stick if it came with one that has the version of LTM that you need on it. This will greatly simplify the upgrade process and get it to at least the major version you need.

Once the unit has been upgraded to at least the major base version that you need. Login via the serial console with root/default and type config. This will let you set the management IP address for the unit.

Once the management address is set, connect to it via the browser with admin/default and start going through the licensing and configuration process.

Upload the hot fixes if necessary to the replacement unit and update to the version needed to restore the backup file. Once the hot fixes are done updating go ahead and restore the backup to the failed unit.

Hookup the failover ethernet cable.

Set backup the HA configuration between the units and ensure that you can ssh between the units on their failover interfaces.

Push the configuration from the Active unit to the new unit with an override, if it fails or there is any issue during the time run this command on the failed unit to see what the issue is:

tmsh show cm sync-status

Once it’s all done and happy it should be back in sync and in an active/standby state.

Then plug in the last of the cables for the internal/external interfaces and then you should be done.

Pack the old unit up and ship it out.

Retrieve SSID info from Meraki Wireless Network

I support multiple Meraki Wireless networks and I wanted the ability to pull in pertinent data from the wireless networks that I support. This will allow me to pull in the data and then upload it into the wireless mapping programs and design software. This will also allow me to easily pull the channels, power settings, SSIDs, and other pertinent information. Here is the link to my GitHub site where I will be keeping the most up to date program as I fix things and work to improve it.

#!/usr/bin/env python

#import necessary modules
import cred
import requests
#Meraki site information
HEADERS = {'X-Cisco-Meraki-API-Key': (cred.key), 'Content-Type': 'application/json'}
#NETWORK = input(str("What network are we looking at? "))
NETWORK_URL = MERAKI_DASHBOARD + '/api/v0/networks/%s/devices' % NETWORK
NETWORK_GET = requests.get(NETWORK_URL, headers=HEADERS)
#Create a function pull in the information
WIRELESS_SETTINGS_URL = MERAKI_DASHBOARD + '/api/v0/networks/%s/devices/%s/wireless/status' % (NETWORK, DEVICE['serial'])
if SSIDS['enabled'] == True:
print("SSID " + (SSIDS['ssidName']) + " BAND " + (SSIDS['band']) + " BSSID " + str(SSIDS['bssid']) + " Channel " + str(SSIDS['channel']) + " Power " + str(SSIDS['power']))
#Loops through the network and the devices to find all of the information.
if DEVICE['model'] == "MR42":
print("AP " + DEVICE['name'])

Passing the CWAP

Last weekend I was able to pass the CWAP exam on my second attempt. The first time through the exam I was thrown by some of the questions and didn’t have as good a grasp on some of the random things that were asked for. So I spent 2 weeks between exam takes watching all of the videos again and then going through all of my notes and flash cards that I had made. I also spent a good amount of time looking at packet traces and figuring out where all of the information and how Transmit Beamforming works with NDP. Overall I felt it was a good exam even though it took me two times to pass it. I definitely feel a lot better now about the information and that I was able to absorb it for the exam. So now I get to use the cool CWAP image for things:Cwap 200x200 My next exam will be the CWSP exam as I work my through to the CWNP.

Here are some of the resources that I used while for the CWAP:
CWNP – Video training
CWNP – Practice Tests
WIFItraining – CWAP Workbook
CWNP – Official CWAP Study Guide

Meraki Script to pull LTE Card Signal

Script for pulling the make and signal strength of wireless cards

We are trying to continually audit our LTE cards in the Meraki Routers so we wanted to be able to monitor the stores LTE connections and see the signal strength and then determine which if any needed to be swapped out. However that data is only stored at the device level so you have to iterate through the whole Organization then by network and then by device in the network. Meraki has a polling limit for how many times you can poll the cloud per second so I put a 1 second delay in there to keep the program from overwhelming everything and causing issues for itself or for our users monitoring on the website.

The script can be found here:

How I handle credentials and shared variables in Python

How to handle common variables between programs

I have been writing a lot of python programs lately for interacting with the Meraki Platform. I was tired of copying and pasting my variables and credentials between programs, plus I wanted the ability to easily upload the programs to GitHub without having to worry about sanitizing the program of my companies or personal data. I did some searching and didn’t find a lot so what I figured I would do is put this information into a python module and then I could call that module from within my programs and then I wouldn’t have to worry about keeping all of my data secure. I decided to call my module and then I could call it from within the program with just a “import cred”. I used to copy this file into each of the directories where I was working on a program. Then I ran into a problem where I had to change an API key, I then had to go through and find all of the files I had created and then update the data in them. That proved to be more of a pain than I wanted to deal with so I decided to place it in a central directory for all of my programs. This proved much easier, but then I had to figure out how to call it from within Python without making it a module in the install path.

That is where I came up with this:

import sys

#Import the CRED module from a separate directory
import cred

With this it allows me to keep one central directory to store all of my credentials, but also commonly needed variables. I call it from within the program and can then run my programs easily. Love to hear how others are handling this or if there is a better way for me to do it.

And now for something completely different – Storage with Brocade

So as attrition has struck and we have lost staff, I am now taking over Storage duties.  It’s a switch right, it should work just like a network switch.  That is the thinking of people above me, I think they have been away from hands on work for too long. I understand iSCSI, that’s simple.  This FC stuff, I have a bit of a curve in front of me.

First order of the day….  Assess where we are and if we need a firmware update to maintain our pci compliance.  Nothing like getting thrown in and going from there.  Well at least I know we have Brocade Swithces so I am starting off with at least one advantage.   So after finding the username and password to log into them, I then needed to find out what version we had.  This website was invaluable in determing that:

If you scroll down to the comments someone has updated with the later versions, but I put the instructions here just as a quick look along with the meanings of the different Switchtypes.

Open web browser at http://hostname of switch/SwitchInfo.html

While I am surprised about the amount of information that is given away for free, I am glad that it is at least available for me to help me out.

Scroll down till you see in List of Ports: switchType:    xx.x

Switch Type Switch Name translation for xx.x
1 Brocade 1000 Switches
2,6 Brocade 2800 Switch
3 Brocade 2100, 2400 Switches
4 Brocade 20×0, 2010, 2040, 2050 Switches
5 Brocade 22×0, 2210, 2240, 2250 Switches
Switch Types and Product Names
Generated by Jive SBS on 2011-01-06-07:00
7 Brocade 2000 Switch
9 Brocade 3800 Switch
10 Brocade 12000 Director
12 Brocade 3900 Switch
16 Brocade 3200 Switch
17 Brocade 3800VL
18 Brocade 3000 Switch
21 Brocade 24000 Director
22 Brocade 3016 embedded Blade Switch
23 8Gbit 10-port embedded fabric switch
26 Brocade 3850 Switch
27 Brocade 3250 Switch
29 Brocade 4012 Embedded Blade Switch
32 Brocade 4100 Switch
33 Brocade 3014 Switch
34 Brocade 200E Switch
36 Brocade FR4-18i Director Blade
37 Brocade 4020 Embedded Blade Switch
38 Brocade 7420 SAN Router
40 Fibre Channel Routing (FCR) Front Domain
41 Fibre Channel Routing (FCR) Xlate Domain
42 Brocade 48000 Director
43 Brocade 4024 Embedded Blade Switch
44 Brocade 4900 Switch
45 Brocade 4016 Embedded Blade Switch
46 Brocade 7500 Switch
Switch Types and Product Names
Generated by Jive SBS on 2011-01-06-07:00
47 Brocade FC4-16IP Director Blade
50 Brocade 4GB FC Port Blade
51 Brocade 4018 Embedded Blade Switch
55 Brocade FA4-18i Extension Director Blade
55,2 Brocade 7600 Switch
58 Brocade 5000 Switch
62 Brocade DCX Backbone
63 Brocade 8Gb Backbone Core Fabric Switch
64 Brocade 5300 Switch
66 Brocade 5100 Switch
67 Brocade Encryption Switch
68 Brocade 8Gb 16 FC 2 GigE ports Director Encryption
69 Brocade 5410 Blade
70 Brocade 8GB 10 Port Embedded Fabric Switch
71 Brocade 300 Switch
72 Brocade 5480 Embedded Blade Switch
75 Brocade M5424 Embedded Blade Switch
76,6 Brocade 8000 FCoE Switch
77,3 Brocade DCX-4S
82 Brocade 8Gb 24-port Embedded Blade Switch
83 Brocade 16-FC port, 6-GE port, auto sensing 1, 2, 4 or
8Gbit Switch
86 Brocade 8Gbit 26-port embedded Switch
88 Brocade 10Gb 24 GigE ports DCE Blade
Switch Types and Product Names
Generated by Jive SBS on 2011-01-06-07:00
89 Brocade 8Gb 12 FC, 1Gb 10 GigE FCIP Blade, 10Gb 2
GigE ports FCR
Now at least I am off and running to the Brocade site to find out exactly how many versions I am back.
I also needed to determine the serial number so as I found out that “?” didn’t do anything for me, I then tried “help” and that listed all of the commands on the switch quite a few.  I figured the command had to start with switch, chassis, or hardware.  I looked through and was able to find this command:
Luckily this gives me everything I need now and can open my support case or get access to the support site to find the information for myself.