Thursday, August 13, 2015

CCIE DC - There's No "Golden Moment"

So I failed my first attempt at the CCIE DC lab last week in RTP.

It was a really odd experience in that we had connectivity issues with the lab almost immediately. Throughout the day access to the CLI was dog slow and at a few points access was completely broken for 10 or 15 minute stretches. The slowness wasn't a show stopper for pure CLI guys (RS candidates), but for Collaboration candidates and DC candidates when they got to UCS, the slowness made graphical interfaces unusable.

David is an interesting guy.
From the sounds of it he's used to people talking bad about him in the blog world; all I'll say is if you don't say a word the whole day and do EXACTLY what he says, you won't be publicly humiliated :D

Once the outages accumulated to two hours David called it a day and we were told to come back the next morning to try again.
It sounded like there was a provider issue causing more widespread issues within Cisco.
The next day access was snappy and there were zero connectivity issues.

I thought the exam was relatively easy. I finished my last task by lunch and spent a couple hours after lunch double and triple checking everything. I won't say exactly what I verified since I'm not sure what will violate the NDA, but suffice it to say I was pretty confident things were working. What I will say, since the precedent has been set in many a blog before me, is that I was able to SAN boot UCS. The proverbial "Golden Moment."

Apparently not.
By my calculations I think I missed passing by 5 points.

I've heard secondhand accounts of both people passing the CCIE DC who did not get their blade to boot, and people failing the lab who did get their blade to boot, and now I'm one.
From seeing how the lab is put together it is easy to see there is no Golden Moment.
After your technical knowledge it's all about attention to detail, correct interpretation, and in my opinion, hoping you chose the correct method the grading script looks for :)

So, back I go, Sep 16.
Happy studies!

Friday, June 26, 2015

Nexus Code Update

New NX-OS code was released June 18:
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/7_x/nx-os/release/notes/72_nx-os_release_note.html#pgfId-812889

Of particular interest is the support for dynamic routing over vPC, FINALLY!!

Though the details are a little confusing to me. This is from the notes:
"The equal routing cost matrices must be configured on applicable interface on each of the vPC peers, failure to do so can result in blocking the traffic. Asymmetric routing feature has to be implemented to address this issue and to configure Dynamic Routing over vPC."

I'm searching for more information on this and I'll updated this post when I get more details.


UPDATE:
I found this updated configuration guide that gives a little more info:
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/interfaces/configuration/guide/b-Cisco-Nexus-7000-Series-NX-OS-Interfaces-Configuration-Guide-Book/configuring-vpcs.html#task_E034B3298DC945D1A33AB27AF73AE88C

I'm scheduled to take my DC lab exam on the 8th, after that I'll see if I can get a lab to explore the configuration.

Friday, April 24, 2015

OTV on ASR with VRF (Multi-homed WAN)

I recently built a virtual datacenter for a customer using the ASR 1001.This one was an interesting  build because we had two WAN connections and a limited amount of hardware to work with. We had to figure out how to deploy OTV on a dual-homed (WAN) ASR router with a total of three interface transceivers.

Here's what we had to work with:

The Problem
The big caveat with OTV is you have to tie an Overlay to a single join interface and it cannot be a loopback interface. How long has that functionality been "planned for a future NXOS release" amirite?
Hoping against hope that Cisco ninja-added it to the latest code I gave it  a try. The weird thing is it let me add a loopback on one router as the join interface, but on the second it threw an error. Then when I went back to the other and removed/re-added it threw an error as well. I think it may have been an order of operations thing where on the first I added the overlay config before I created the loopback and reversed the order on the second router. Anyway, no go.

Another minor problem as you can see from the drawing above is port count. I had three physical ports and I needed the following:
  • An MPLS Port
  • A Metro E Port
  • A join interface
  • A "inside" routed interface for L3 traffic
  • "Inside" trunk/tagged links for the OTV VLANs
The Solution
I'll address the port count first since that's an easy fix, but first here's the physical drawing of how the transceivers were utilized:
The good news is a physical interface can be configured with both service instances for OTV-bridged traffic as well as traditional dot1q sub-interfaces. So we simply configure the inside interface (g0/0/0) with the necessary service instances as well as a dot1q sub-interface for the inside L3 connectivity. Here's what the Overlay and g0/0/0 configurations look like up to this point (All IP addressing has been changed):

interface Overlay1
 no ip address
 !
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 101 ethernet
  encapsulation dot1q 101
  bridge-domain 101
 !
interface GigabitEthernet0/0/0
 description Inside Trunk
 mtu 9100
 no ip address
 negotiation auto
 service instance 1 ethernet
  encapsulation untagged
  bridge-domain 1
 !
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 101 ethernet
  encapsulation dot1q 101
  bridge-domain 101
 !
!
interface GigabitEthernet0/0/0.10
 description Inside
 encapsulation dot1Q 10
 ip address 192.168.10.2 255.255.255.248
 ip nat inside
 zone-member security INSIDE
 ip ospf 1 area 0
!

Now for the join interface. Since we can't use a loopback, and we can't use our multiple WAN ports, I decided to create another sub-interface on the inside port to bounce traffic off the 4500. To keep the traffic clean/separate/symmetric, I made a VRF just for the join interface.
Here's the full overlay and inside interface configuration:
interface Overlay1
 no ip address
 otv join-interface GigabitEthernet0/0/0.11
 otv use-adjacency-server 192.168.11.1 unicast-only
 otv adjacency-server unicast-only
 service instance 10 ethernet
  encapsulation dot1q 10
  bridge-domain 10
 !
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 101 ethernet
  encapsulation dot1q 101
  bridge-domain 101
 !
interface GigabitEthernet0/0/0
 description Inside Trunk
 mtu 9100
 no ip address
 negotiation auto
 service instance 1 ethernet
  encapsulation untagged
  bridge-domain 1
!
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 101 ethernet
  encapsulation dot1q 101
  bridge-domain 101
 !
!
interface GigabitEthernet0/0/0.10
 description Inside
 encapsulation dot1Q 127
 ip address 192.168.10.2 255.255.255.248
 ip nat inside
 zone-member security INSIDE
 ip ospf 1 area 0
!
interface GigabitEthernet0/0/0.11
 description OTV Join
 encapsulation dot1Q 128
 ip vrf forwarding otv_join
 ip address 192.168.11.1 255.255.255.248
!

And here's what the traffic flow would look like between two OTV hosts:

Thursday, March 5, 2015

North/South Firewalling in Active/Active Virtual Datacenters

I recently had the opportunity to design and build an active/active virtual datacenter for a customer. The design presented some interesting challenges when it came to firewalling and routing in general such as how to minimize traffic tromboning and how to deal with asymmetric traffic flows.
The following is a walkthrough of the solution design and implementation.

The Sandbox

Here is a high-level drawing of the equipment we had to work with and the site connectivity:



The Requirements, Challenges, & Strategy
We noted the following technology drivers/requirements for our datacenter design:
1.       Support VM mobility keeping IPs the same
2.       Traffic should take the shortest path possible (minimize tromboning & sub-optimal routing) whenever possible including to the first hop gateway (local egress routing)
3.       Fault isolation. To the extent possible, issues at one campus datacenter should not affect the other campus datacenter (Separate Spanning Tree Domains)
4.       Building redundancy. We should be able to lose one building/datacenter entirely and still have access to the remaining datacenter’s servers
5.       Firewalling. North/South traffic should pass through a firewall even if a complete building fails. Additionally in this case the requirement was to have the firewalls physically in-line separating user space and server space.
6.       Customer wanted Server East/West traffic to stay on the server access layer 5Ks and not have to flow up to the user space/7Ks

Based on the requirements we chose a few technologies & strategies for networking:
·         L3 connections between the campus buildings to isolate Spanning Tree domains
·         OTV between datacenters for VM mobility & FHRP localization
·         OTV will ride L3 connections between the 5Ks so East/West traffic doesn’t have to flow up to the user block
·         Distributed firewalling for building redundancy
·         VDC firewall sandwich for in-line firewalling

These solutions introduce a new challenge. Traffic to and from OTV networks will flow asymmetrically in some cases especially if there are no ingress routing enhancements such as LISP or Global Server Load Balancing. If we assume at some point we will experience asymmetric traffic flows, standalone firewalls are not a viable option.
A distributed Active/Standby firewall was not desire because of sub-optimal routing (requirement #2).

To address this we decided to use the ASA clustering capability of the 5585X firewall. ASA clustering allows active/active firewalling and supports asymmetric traffic flows. Return flows that land on the wrong firewall are sent over a cluster link to the correct firewall.
We will still experience some sub-optimal routing in the case of asymmetric traffic flows, but with proper network design we should be able to greatly minimize it.

The Solution
User Block
The user blocks at each site consist of the Nexus 7K default VDCs. Closet switches will be aggregated here as well as other services blocks (internet edge, WAN routing, etc).
The user blocks at each site were connected via two of the eight dark fiber pairs.
The link is a trunk link that carries the OSPF VLAN and a few other vlans that need to be stretched in the user block. We decided not to use OTV here since the vlans will be minimized and they aren’t super important (a Spanning Tree issue that took down the vlan wouldn’t be catastrophic).

Here’s the topology:

Firewall VDC Sandwich
Each firewall has four ten gig interfaces we can use. We need one inside, one outside, and since the cluster link is pretty important we will etherchannel those.
To make the firewall physically in-line, and also to aggregate the multiple 5K uplinks we used a VDC sandwich topology.

Here’s the topology:


The Cluster Link is spanned between the two sites at the user block (7Ks). This breaks our rule of stretching Spanning Tree between the two datacenters, but to maximize stability we make sure the STP instance is only on the two 7Ks and only on this datacenter interconnect (DCI). This can be achieved by only creating the cluster link vlan on the two 7Ks and/or manually pruning the vlan off all links except the DCI.

The Inside & Outside interfaces need to be on the same subnet for ASA clustering, but we did not want to stretch vlans between the two datacenters. I decided to create the vlans/subnets in both datacenters, but they are disjointed (not connected at layer 2).
This design gives us two big benefits:
·         No spanning additional vlans across the user block DCI or having to connect the ServerAgg VDCs to span the inside interface vlans.
·         Local ingress/egress routing. Since we don’t have an equal cost path to the other datacenter’s firewall (which would be the case if the firewall interfaces were merged at layer 2), we will always take the direct/local path in or out.
The second bullet is big, it is the key concept of this design. Let’s take a look at the Outside (North/Yellow) interfaces. If those two networks were merged with a vlan across the DCI, southbound traffic would have two equal cost paths (assuming a dynamic routing protocol is in use). One local and one across the DCI. This would introduce sub-optimal/asymmetric routing as shown in the following illustration:

Severing the layer 2 link between the interfaces fixes this:


At first I was not sure if this was possible. I studied the ASA Clustering design guide multiple times. Though the guide assumes firewalls in the same datacenter, I didn’t see any technology reason they had to be. To be safe I had a phone call with my local Cisco rep and another Cisco engineer from the appropriate BU who confirmed that’s how you would do it. To be safe he wanted to run it by some of his colleagues and a couple weeks later I had solid confirmation it was a valid design.

In general having the same disjointed subnet at two different locations is bad because you would only be able to route to the one local or closest to you (think Anycast RP). However, two key principles allow us to do it:
1.       ASA clustering has no interface monitoring directly between the non-cluster link interfaces. Meaning the two outside interfaces don’t have to ping each other.
2.       The Inside and Outside interfaces are transit links only. Traffic never has to be directed to the interfaces themselves (besides routing protocol hellos but those are directly connected).

#2 is sort of true, we have to consider management.
Usually we manage ASA firewalls directly by the inside (or outside) interface.
We do have a dedicated management port, but that port is really meant for out-of-band management. If we send traffic to the management port, return traffic would want to exit the data link (inside or outside interface depending on the source).
Assuming we don’t have an out-of-band management network we would only be able to reach the firewall we’re local or closest to for management purposes.
Here’s an illustration of the problem:


Assuming we don’t want to mess with static host routes or other router tricks, we considered three options to address this issue:
1.       A jump host on the management VLAN
2.       A console server
3.       Multi-context
Options 1 & 2 are really only viable if administrative access is the only directed traffic to the ASA, but we also had to consider management platforms. SNMP polling engines would run into the same problem, therefore we chose #3.

With multi-context we can assign a single management interface to the admin context. We would stretch that vlan over the DCI. It could be OTV, but we chose to do it over the user block DCI since it’s not a super important VLAN and like the cluster link vlan we can prune it down.
Here’s what it looks like:


The cluster link is gone in this drawing because it resides in the system context. With this setup we can now route to and from the management interface with no problems.

Server Block
For the server block we used the Nexus 5Ks as the first hop gateway for all server subnets. The 5Ks connect up to the ServerAgg VDC on the 7K to funnel in to the firewall inside interface.
OTV runs across the 5Ks for East/West server traffic within the same VLAN. HSRP filters are configured in the OTV VDC for first hop localization.
The Server Block DCI is pure layer 3 (no spanned VLANs) using 4 pairs of fiber.
Here’s what it looks like:


Traffic Flows
This section is a walkthrough of some common traffic flows to illustrate how the topology will handle the traffic.

Flow #1: Client to OTV or Non-OTV server in same datacenter

Flow #2: Client to Non-OTV server in remote datacenter

A quick note here. In this topology if all our links are the same speed we will have an equal cost path at step 3 West across the 5Ks. This can be addressed by either a port-channel in the datacenter or on the User block DCI (Don’t forget to use the bandwidth statement on vlan interfaces) to lower the OSPF cost, or manually increasing the OSPF cost on the Server block DCI, or the way we did it was to put everything south of the 7Ks in a dedicated OSPF Totally Stubby Area. This originates a default route into the area so that for all egress routing we simply take the path to the closest area border router (ABR) which would be the local ABR.

Flow #3: Client to OTV server in remote datacenter (Assuming no LISP)

Flow #4: Remote sites
If remote sites or buildings are dual-homed to the two datacenters they will have an equal cost path to OTV VLANs. Depending on which datacenter they land on and which specific host they’re going to the flow will look like Flow #1 or Flow #3.

Enhancements
I’m sure there is plenty of room for enhancement to this design. We could deploy LISP or GSLB to achieve more symmetrical routing. Maybe leverage OTV or a dedicated DCI switch for vlans that need to span the user block. I’d love to hear any ideas you have.

East/West Firewalling
This design is specifically meant to address North/South firewalling. For East/West firewalling there are multiple options that would fit in nicely. Some are:
·         Connect L2 links from the 5Ks to the Agg VDCs and break off DMZs in the ASA with sub-interfaces on the inside interface.
·         VMware NSX Distributed firewall
·         Cisco FirePOWER NGIPSv

Tuesday, February 10, 2015

SDN Killed the Routing Star (a newcomers thoughts)

If you’re like I was, the term “SDN” hits your ears like a vuvuzela during a soccer match. And then there’s the rest of the title; I can almost hear a collective rolling of the eyes from my fellow Network Engineers. To be clear I don’t foresee the need for talented Route/Switch Network Engineers ever going away, but I do think their role might change and I think it’s due to another buzzword we’ve heard: Consumption model.

Until recently I’ve always thought SDN was only for large multi-tenant service providers or organizations with large Test/Dev environments that required rapid, user-driven provisioning of network resources. Since that is a relatively small market, I figured SDN is a passing fad that will never affect me or my customers.
Let’s take a closer look at this article’s title. Yes, it is a spoof of the late 1970s music video titled “Video Killed the Radio Star.” This is a song about the ushering in of a new era, about how music videos would render useless the radio star. Why? Because a change in how audiences consumed music re-defined musicians. Before music video, musicians were defined audibly, by how they sounded over the radio because that’s how music was consumed. Once music began to be consumed primarily over television, suddenly the musician was defined not only audibly, but visually.

A face made for radio.
Let’s face it, if vocal talent is all a musician has today, his or her chances of success are drastically lower than the same vocal talent with visual appeal and great showmanship.
Again, changes in the consumption model of music drove a re-definition of the musician.
What drove this point home for me was a story I heard in a VMware NSX class:
Our instructor spent a lot of his time in a particular datacenter. He would frequently walk by a cage with a unique set of server racks. These server racks were more shelves than racks, and he described the servers as cheap bare (no chassis) motherboards sitting on the shelves with a few connections to power, network, and a ribbon cable to a hard drive. He thought to himself “who would do a datacenter like this?” One day he saw a technician replacing a hard drive in the cage (no surprise there with that kind of setup) and struck up a conversation. The technician revealed the cage belonged to tech giant, Google.
Google, with their army of developers had written a distributed computing platform that completely changed how they consumed compute resources. To Google, the power of compute was no longer defined by hardware specifications, but by software, and so the underlying hardware experienced a change in focus. Notice I paid special attention to not say “became less important.”
I believe networks are on the cusp of this same change in consumption model and unlike the aesthetically-challenged musician, we (Network Engineers) can adapt!

A change in consumption model drives SDN, enabled by Microsegmentation and Overlays.
I’d like to start this section by exploring a couple network challenges in the modern datacenter:
·         Network Topology- A Cisco rep once told me a loooooong time ago, “route where you can, switch where you must.” This was the mantra of more simple times. Today with the proliferation of compute virtualization and features therein such as VM mobility (vMotion), the industry is driving network topologies back to a more flat design. However, we all know the drawbacks of large layer 2 domains.
·         Firewalling in the distributed datacenter- In addition to supporting VM mobility across datacenters on a flat network, we also want to protect our servers from the corporate network as well as protect the various application tiers from each other. This presents a challenge to network engineers to properly steer traffic to a set of firewalls while maintaining datacenter redundancy and traffic flow symmetry/optimized routing.
Solutions for these challenges include network overlays and microsegmentation.

Overlays.
In overly-simplified terms, network overlays allow the network engineer to create network topologies on top of and completely separate from the underlying physical network topology.
This gives us the ability to maintain the old mantra on the underlay and realize all the benefits thereof, but also create virtual flat networks on top so that network consumers can realize the benefits of that kind of topology.

Microsegmentation.
With overlay networks alone we still have (and it actually compounds) the issue of datacenter firewalling.
Microsegmentation is the solution. A distributed firewall shimmed between a virtual machine’s virtual NIC and its virtual network port. We no longer need to steer traffic to a firewall appliance or worry about traffic symmetry because we effectively have a single gigantic transparent firewall where every host is plugged into its own dedicated firewall interface.

Network Consumption Model.
With the solutions noted, we can see how the network consumption model will change. With overlays and microsegmentation, suddenly the network no longer needs to be a giant complicated mess of firewalls and routing policies. We need a solid, redundant, simple (relatively) network configuration for establishing physical connectivity. Ideally the physical network would rarely change except to add scale or upgrade aging hardware.
The overlay is where the real intelligence resides and because it is implemented in software it is extremely flexible. It can change rapidly and programmatically as required by the consumers.

Fate of the Routing Star.
Some network engineers may fear SDN and wonder where they fit in if the <insert virtualization provider here> guys take over networking. From the vision I’ve laid out we can clearly see the continued need for network engineers. Somebody has to design, build, and maintain the overlay networks. Somebody has to build the auto-provisioning workflow. Underneath it all there is still data being wrapped in IP packets. Wrapped in frames. Tagged with headers. Wrapped in another IP packet. Wrapped in another frame…
I submit that the best man/woman for the job is still a network engineer.


Nexus Fu Re-Purpose

Obviously I've been slacking with my CCIE DC studies and blog posts. It usually takes a CiscoLive (or the like) to re-motivate me to study and blog. This time is no different, except this time it was a VMware Partner Exchange conference.
I had the opportunity to take the NSX Install Configure Manage boot camp right before the conference. I attended several good NSX sessions, and was able to take and pass the VCP-NV exam. I'm pretty excited now about NSX and SDN in general. I'm still going to study for my CCIE DC (probably slower than I should) and continue to blog Nexus scenarios, but I want to expand the scope of my blogs to include all datacenter technologies that interest me. So in addition to Nexus blogs I plan on exploring UCS, NSX, and whatever else interests me in the Datacenter realm.

Wednesday, November 13, 2013

FabricPath - Port Channels & ISIS Cost

I was doing a small FabricPath implementation a while back and ran into a couple things I thought I would blog about.
One is regarding the use of port channels in a FabricPath topology and the other is ISIS routing when vPC peers use smaller links than the uplink ports (I'll explain that better in a bit).

To start, here's the topology I'm using for the lab:


To port channel, or not to port channel???
FabricPath allows us to create arbitrary topologies with "lots of links" maintaining 100% forwarding topologies with ISIS's inherent equal cost multipathing (ECMP) feature. So why would we consider port channels?
We certainly don't have to, but the reason we may consider it has to do with how FabricPath handles multidestination traffic and scaling to a smaller degree.

I'll start with scaling since it's the easiest to cover. FabricPath allows a maximum of 16 equal cost paths. If we need to scale beyond this (maybe for bandwidth reasons????) we would have to use port channels. Since a port channel counts as a single FabricPath link, we can scale physical paths exponentially.

In my opinion, multidestination trafic is a bigger factor. Multidestination traffic (multicast, unknown unicast) does not use equal cost multipathing. There are Cisco documents that explain it well, but a quick explanation is a root bridge is selected to create a "multidestination tree," then a single path is selected towards the root for a given multidestination traffic flow.

So if you have a lot of multidestination traffic, port-channels can provide additional bandwidth that couldn't be utilized with FabricPath ECMP alone.

Let's take a look at this on the CLI. I'm going to jump on N5K-1 in the topology above and first see how we would route normally towards the virtual switch ID for the 7K pair, 170.
N5K-p8-1(config)# show fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id


FabricPath Unicast Route Table for Topology-Default

0/151/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 00:44:59, local
1/150/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 00:38:49, local
1/152/0, number of next-hops: 1
        via Po1, [115/20], 0 day/s 00:38:49, isis_fabricpath-default
1/170/0, number of next-hops: 8
        via Eth1/1, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/2, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/3, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/4, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/5, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/6, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/7, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/8, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
1/171/0, number of next-hops: 4
        via Eth1/1, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/2, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/3, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/4, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
1/172/0, number of next-hops: 4
        via Eth1/5, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/6, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/7, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
        via Eth1/8, [115/40], 0 day/s 00:36:27, isis_fabricpath-default
2/150/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 00:38:49, local
N5K-p8-1(config)#
So we see from the output that we have eight equal cost paths to switch 170 (4 each to 171 & 172).
Now lets see what multidestination trees we have and what path they're using:
N5K-p8-1(config)# sh fabri isis trees
Fabricpath IS-IS domain: default
Note: The metric mentioned for multidestination tree is from the root of that tr
ee to that switch-id

MT-0
Topology 0, Tree 1, Swid routing table
152, L1
 via Ethernet1/4, metric 40
170, L1
 via Ethernet1/4, metric 20
171, L1
 via Ethernet1/4, metric 0
172, L1
 via Ethernet1/4, metric 20

Topology 0, Tree 2, Swid routing table
150, L1
 via Ethernet1/8, metric 40
152, L1
 via Ethernet1/8, metric 40
170, L1
 via Ethernet1/8, metric 20
171, L1
 via Ethernet1/8, metric 20
172, L1
 via Ethernet1/8, metric 0

N5K-p8-1(config)#
We have two trees. Based on the zero metric, we can see N7K-1 is the root for one, N7K-2 is the root for the other. We can see that each tree is using a single physical path to the root (1/4 & 1/8).

Now I've converted the uplinks between the 5Ks & 7Ks to port channels so that the new topology is:
Let's jump back on N5K-1 and see how the normal FabricPath routing looks for routing toward 170:

N5K-p8-1(config)# sh fabri route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id


FabricPath Unicast Route Table for Topology-Default

0/151/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 01:08:39, local
1/150/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 01:02:29, local
1/152/0, number of next-hops: 3
        via Po1, [115/20], 0 day/s 01:02:29, isis_fabricpath-default
        via Po171, [115/20], 0 day/s 00:09:55, isis_fabricpath-default
        via Po172, [115/20], 0 day/s 00:01:11, isis_fabricpath-default
1/170/0, number of next-hops: 2
        via Po171, [115/10], 0 day/s 00:10:18, isis_fabricpath-default
        via Po172, [115/10], 0 day/s 00:01:11, isis_fabricpath-default
1/171/0, number of next-hops: 1
        via Po171, [115/10], 0 day/s 00:10:18, isis_fabricpath-default
1/172/0, number of next-hops: 1
        via Po172, [115/10], 0 day/s 00:01:11, isis_fabricpath-default
2/150/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 01:02:29, local
N5K-p8-1(config)#
We see now that we have two equal cost paths through port channels 171 & 172 (one link each for switch id 171 & 172)

Now lets take a look again at the multidestination trees:
N5K-p8-1(config)# sh fabri isis trees
Fabricpath IS-IS domain: default
Note: The metric mentioned for multidestination tree is from the root of that tr
ee to that switch-id

MT-0
Topology 0, Tree 1, Swid routing table
152, L1
 via port-channel171, metric 10
170, L1
 via port-channel172, metric 20
171, L1
 via port-channel171, metric 0
172, L1
 via port-channel172, metric 20

Topology 0, Tree 2, Swid routing table
150, L1
 via port-channel172, metric 10
152, L1
 via port-channel172, metric 10
170, L1
 via port-channel172, metric 20
171, L1
 via port-channel172, metric 20
172, L1
 via port-channel172, metric 0

N5K-p8-1(config)#
Here again a single link is selected but now its a logical link (the port channel) representing multiple physical links, so we get more bandwidth for our multidestination traffic.

FabricPath ISIS Cost With Port Channels and Smaller vPC Peer Links
I couldn't think of a clever, shorter heading to describe the scenario, but here we go.

Configuring the port channels has uncovered a new, interesting issue. Go back up real quick and compare the regular FabricPath routing tables, specifically from N5K-1 to its peer switch (152).
You'll see that in the first output it had a single path (po1) directly to its peer, which is what we expect.
However, after adding the port channels up to the 7Ks we now see three equal cost paths to our neighbor!!!

Generally, we design vPC Peer Links with less bandwidth because it normally doesn't carry much traffic. Usually my designs end up with exactly half the bandwidth between the peers as the uplinks to the external switches, which is the case in this lab. The result is an equal cost path to our peer, through the external switch. The vPC peer link has an ISIS cost of 20. The uplink to the external switch has an ISIS cost of 10, and its uplink to my peer has a cost of 10 as well.
We might get away with this down at the 5Ks where we're pure layer 2, but up at the 7Ks where I have an EIGRP peering in a VLAN carried over the vPC peer link it wreaks havoc.

Let's verify the issue on N7K-1:
N7K-8-1(config)# sh ip ei nei
IP-EIGRP neighbors for process 1 VRF default

N7K-8-1(config)#
N7K-8-1(config)#
N7K-8-1(config)#
N7K-8-1(config)#
N7K-8-1(config)# sh fabri route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id


FabricPath Unicast Route Table for Topology-Default

0/171/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 01:41:55, local
1/150/0, number of next-hops: 2
        via Po151, [115/10], 0 day/s 00:31:42, isis_fabricpath-default
        via Po152, [115/10], 0 day/s 00:31:19, isis_fabricpath-default
1/151/0, number of next-hops: 1
        via Po151, [115/10], 0 day/s 00:31:42, isis_fabricpath-default
1/152/0, number of next-hops: 1
        via Po152, [115/10], 0 day/s 00:31:19, isis_fabricpath-default
1/170/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 01:36:34, local
1/172/0, number of next-hops: 3
        via Po1, [115/20], 0 day/s 01:36:34, isis_fabricpath-default
        via Po151, [115/20], 0 day/s 00:22:34, isis_fabricpath-default
        via Po152, [115/20], 0 day/s 00:30:24, isis_fabricpath-default
2/170/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 01:36:34, local
N7K-8-1(config)#
First I've verified that my EIGRP adjacency is in fact dead. Next I look at the FabricPath routing table and I can see the three equal cost paths to my peer switch (172).

I haven't seen where Cisco provides the why, but their recommendation is to use dedicated L3 links for routing protocol adjacencies. I suspect this might be at least part of the reason.
But let's see if we can fix this without having to change our physical topology.

I'm going to statically assign an ISIS cost of 10 to our port channel:
N7K-8-1(config)# int po1
N7K-8-1(config-if)# fabri isis metric 10
N7K-8-1(config-if)# exit
N7K-8-1(config)#
N7K-8-1(config)#
N7K-8-1(config)# sh fabri route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id


FabricPath Unicast Route Table for Topology-Default

0/171/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 01:47:06, local
1/150/0, number of next-hops: 2
        via Po151, [115/10], 0 day/s 00:36:53, isis_fabricpath-default
        via Po152, [115/10], 0 day/s 00:36:30, isis_fabricpath-default
1/151/0, number of next-hops: 1
        via Po151, [115/10], 0 day/s 00:36:53, isis_fabricpath-default
1/152/0, number of next-hops: 1
        via Po152, [115/10], 0 day/s 00:36:30, isis_fabricpath-default
1/170/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 01:41:45, local
1/172/0, number of next-hops: 1
        via Po1, [115/10], 0 day/s 01:41:45, isis_fabricpath-default
2/170/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 01:41:45, local
N7K-8-1(config)#
N7K-8-1(config)#
N7K-8-1(config)#
N7K-8-1(config)#
N7K-8-1(config)#
N7K-8-1(config)# sh ip ei nei
IP-EIGRP neighbors for process 1 VRF default
H   Address                 Interface       Hold  Uptime  SRTT   RTO  Q  Seq
                                            (sec)         (ms)       Cnt Num
0   10.0.99.2               Vlan999         11   00:00:17  1    5000  1   0
N7K-8-1(config)#
And we're fixed. We have a single path over the vPC peer link and we can see the EIGRP adjacency is back online.

So there are a couple considerations for you when you're designing FabricPath topologies. I've heard recommendations both ways, not to use port channels & to use them with FabricPath. I still haven't decided whether using them is worth it, but I'm leaning towards not using them unless there's a strong case for added multidestination bandwidth.