17 novembre 2014

Enhanced VPC with Fex

eVPC requires asic-specific support, so it is currently available only on the 5500 platforms (5548P, 5548UP, 4496UP).

Whereas before we had to make a decision between two fabric extender (fex) topologies, VPC/full mesh fex uplinks for active/standby server connections or single-uplink/U topologies for active/active server connections, eVPC allows us to build one topology (VPC/full mesh) that supports both active/standby and active/active server connections.

Here's the topology we will be configuring the Nexus switches to support



VPC is somewhat sensitive to the order of operations during configuration. The order of our configuration will be as follows:
1. Establish a keepalive link
2. Establish the port channel between the peer switches
3. Configure the VPC domain
4. Configure FEX ports and add to VPC
5. Configure host ports

1. Establish a keepalive link:


N5K-1#
N5K-1# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
N5K-1(config)# int mgmt0
N5K-1(config-if)# ip add 192.168.1.181/30
N5K-1(config-if)#
N5K-1(config-if)#


N5K-2#
N5K-2# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
N5K-2(config)# int mgmt0
N5K-2(config-if)# ip add 192.168.1.182/30
N5K-2(config-if)#
N5K-2(config-if)# ping 192.168.1.181 vrf management
PING 192.168.1.181 (192.168.1.181): 56 data bytes
Request 0 timed out
64 bytes from 192.168.1.181: icmp_seq=1 ttl=254 time=1.094 ms
64 bytes from 192.168.1.181: icmp_seq=2 ttl=254 time=0.549 ms
64 bytes from 192.168.1.181: icmp_seq=3 ttl=254 time=0.535 ms
64 bytes from 192.168.1.181: icmp_seq=4 ttl=254 time=0.537 ms

--- 192.168.1.181 ping statistics ---
5 packets transmitted, 4 packets received, 20.00% packet loss
round-trip min/avg/max = 0.535/0.678/1.094 ms
N5K-2(config-if)#

2. Now let's establish the etherchannel link between the two switches:


N5K-1(config-if)#
N5K-1(config-if)# feature lacp
N5K-1(config)# int e1/17-18
N5K-1(config-if-range)# switchport
N5K-1(config-if-range)# channel-group 1 mode active
N5K-1(config-if-range)#
N5K-1(config-if-range)# int po1
N5K-1(config-if)# switchport mod trunk
N5K-1(config-if)# spanning-tree port type network


N5K-2(config-if)#
N5K-2(config-if)# feature lacp
N5K-2(config)# int e1/17-18
N5K-2(config-if-range)# channel-group 1 mode active
N5K-2(config-if-range)# int po1
N5K-2(config-if)# switchport mode trunk
N5K-2(config-if)# spanning-tree port type network

3. Configure the VPC domain:


N5K-1(config-if)#
N5K-1(config-if)# feature vpc
N5K-1(config)# vpc domain 1
N5K-1(config-vpc-domain)# peer-keepalive destination 192.168.1.182 source 192.168.1.181 vrf management
N5K-1(config-vpc-domain)#
N5K-1(config-vpc-domain)# int po1
N5K-1(config-if)# vpc peer-link
Please note that spanning tree port type is changed to "network" port type on vPC peer-link.
This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance
(which is enabled by default) is not disabled.
N5K-1(config-if)#


N5K-2(config-if)#
N5K-2(config-if)# feature vpc
N5K-2(config)# vpc domain 1
N5K-2(config-vpc-domain)# peer-keepalive destination 192.168.1.181 source 192.168.1.182 vrf management
N5K-2(config-vpc-domain)#
N5K-2(config-vpc-domain)# int po1
N5K-2(config-if)# vpc peer-link
Please note that spanning tree port type is changed to "network" port type on vPC peer-link.
This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance
(which is enabled by default) is not disabled.
N5K-2(config-if)#

Before we move on to configuring the FEXes, let's do some verification of the trunks, port-channel, and VPC:


N5K-1(config-if)#
N5K-1(config-if)# sh int trunk

--------------------------------------------------------------------------------
Port          Native  Status        Port
              Vlan                  Channel
--------------------------------------------------------------------------------
Eth1/17       1       trnk-bndl     Po1
Eth1/18       1       trnk-bndl     Po1
Po1           1       trunking      --

--------------------------------------------------------------------------------
Port          Vlans Allowed on Trunk
--------------------------------------------------------------------------------
Eth1/17       1-3967,4048-4093
Eth1/18       1-3967,4048-4093
Po1           1-3967,4048-4093

--------------------------------------------------------------------------------
Port          Vlans Err-disabled on Trunk
--------------------------------------------------------------------------------
Eth1/17       none
Eth1/18       none
Po1           none

--------------------------------------------------------------------------------
Port          STP Forwarding
--------------------------------------------------------------------------------
Eth1/17       none
Eth1/18       none
Po1           1

--------------------------------------------------------------------------------
Port          Vlans in spanning tree forwarding state and not pruned
--------------------------------------------------------------------------------
Eth1/17       --
Eth1/18       --
Po1           --

--------------------------------------------------------------------------------
Port          Vlans Forwarding on FabricPath
--------------------------------------------------------------------------------
N5K-1(config-if)#

I have no vlans assigned or vlan interfaces created so there are no vlans in forwarding state, but the trunk otherwise appears healthy..


N5K-1(config-if)#
N5K-1(config-if)# sh port-channel summary
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SU)     Eth      LACP      Eth1/17(P)   Eth1/18(P)  
N5K-1(config-if)#

Healthy port channel...


N5K-1(config-if)#
N5K-1(config-if)# sh vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 1  
Peer status                     : peer adjacency formed ok     
vPC keep-alive status           : peer is alive                
Configuration consistency status: success
Per-vlan consistency status     : success                      
Type-2 consistency status       : success
vPC role                        : primary                      
Number of vPCs configured       : 0  
Peer Gateway                    : Disabled
Dual-active excluded VLANs      : -
Graceful Consistency Check      : Enabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans   
--   ----   ------ --------------------------------------------------
1    Po1    up     1                                                        
N5K-1(config-if)#

...and a healthy VPC. It's worth noting at this point why order of operations is important. If a keepalive link is not established, the VPC will not come online. VPC can and will operate in the event a keepalive link goes down, but the first time the VPC initializes, the keepalive link is required or it will fail. You will see the following output from a "sh vpc"


N5K-1#
N5K-1# sh vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 1  
Peer status                     : peer link is down            
                                  (peer-keepalive not operational,          
                                  peer never alive)                         
vPC keep-alive status           : Suspended (Destination IP not reachable)
Configuration consistency status: failed 
Per-vlan consistency status     : success                      
Configuration consistency reason: Consistency Check Not Performed
Type-2 consistency reason       : Consistency Check Not Performed
vPC role                        : primary                      
Number of vPCs configured       : 0  
Peer Gateway                    : Disabled
Dual-active excluded VLANs      : -
Graceful Consistency Check      : Enabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans   
--   ----   ------ --------------------------------------------------
1    Po1    up     -                                                        
N5K-1#

Compare this to the output when the VPC has been brought online successfully and we later lose the keepalive link:


N5K-1(config-if)#
N5K-1(config-if)# sh vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 1  
Peer status                     : peer adjacency formed ok     
vPC keep-alive status           : peer is not reachable through peer-keepalive
Configuration consistency status: success
Per-vlan consistency status     : success                      
Type-2 consistency status       : success
vPC role                        : primary                      
Number of vPCs configured       : 0  
Peer Gateway                    : Disabled
Dual-active excluded VLANs      : -
Graceful Consistency Check      : Enabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans   
--   ----   ------ --------------------------------------------------
1    Po1    up     1                                                        
N5K-1(config-if)#


4. Configure FEX ports and add to VPC

Let's move on to the FEXes. We will configure two FEXes, 101 & 102. We will do 101 first. We simply add all the uplinks connected to FEX 101 on both 5Ks to the same port channel, and enable VPC. Also, I'll display the output of a "show fex" after each step so we can see how the FEX transitions through states as we configure the ports. First, lets enable the feature and see how the FEXes respond:


N5K-1(config-if)#
N5K-1(config-if)# sh fex
                       ^
% Invalid command at '^' marker.
N5K-1(config-if)#
N5K-1(config-if)#
N5K-1(config-if)# feature fex
N5K-1(config)# sh fex
  FEX         FEX           FEX                       FEX              
Number    Description      State            Model            Serial    
------------------------------------------------------------------------
---       --------            Discovered     N2K-C2248TP-1GE   SSI14280VS8
---       --------            Discovered     N2K-C2248TP-1GE   SSI14280VXT
N5K-1(config)#

So once we enable the fex feature we can issue the "sh fex" command and at this point the 5K sees the FEXes attached, but it's not doing anything with them for now (discovered). For the next step we will configure the uplinks in a port channel and tell the switch these ports are a Fabric Extender:


N5K-1(config-if)#
N5K-1(config)# int e1/7-8
N5K-1(config-if-range)# switchport
N5K-1(config-if-range)# channel-group 101 mode on
N5K-1(config-if-range)# int po101
N5K-1(config-if)# switchport mode fex-fabric
N5K-1(config-if)# vpc 101
N5K-1(config-if)#
N5K-1(config-if)# sh fex
  FEX         FEX           FEX                       FEX              
Number    Description      State            Model            Serial    
------------------------------------------------------------------------
---       --------            Discovered     N2K-C2248TP-1GE   SSI14280VS8
---       --------            Discovered     N2K-C2248TP-1GE   SSI14280VXT
N5K-1(config-if)#

With FEXes we have a new switchport mode called "fex-fabric." We see the state is still "Discovered" at this point. Let's now assign the port-channel a FEX ID:


N5K-1(config-if)#
N5K-1(config-if)# int p101
N5K-1(config-if)# fex associate 101
N5K-1(config-if)#
N5K-1(config-if)# sh fex
  FEX         FEX           FEX                       FEX              
Number    Description      State            Model            Serial    
------------------------------------------------------------------------
101        FEX0101               Offline     N2K-C2248TP-1GE   SSI14280VS8
---       --------            Discovered     N2K-C2248TP-1GE   SSI14280VXT
N5K-1(config-if)#
N5K-1(config-if)# sh fex
  FEX         FEX           FEX                       FEX              
Number    Description      State            Model            Serial    
------------------------------------------------------------------------
101        FEX0101            Registered     N2K-C2248TP-1GE   SSI14280VS8
---       --------            Discovered     N2K-C2248TP-1GE   SSI14280VXT
N5K-1(config-if)#
N5K-1(config-if)#
2012 Jan 28 07:28:32 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SATCTRL_FEX_MISCONFIG: FEX-100 is being configured as 101 on different switch
2012 Jan 28 07:30:17 N5K-1 %$ VDC-1 %$ last message repeated 2 times
2012 Jan 28 07:30:17 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_ENV_ERROR: FEX-101 Module 1: Check environment alarms.
2012 Jan 28 07:30:22 N5K-1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online
2012 Jan 28 07:30:22 N5K-1 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line
2012 Jan 28 07:30:26 N5K-1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

N5K-1(config-if)#
N5K-1(config-if)# sh fex
  FEX         FEX           FEX                       FEX              
Number    Description      State            Model            Serial    
------------------------------------------------------------------------
101        FEX0101                Online     N2K-C2248TP-1GE   SSI14280VS8
---       --------            Discovered     N2K-C2248TP-1GE   SSI14280VXT
N5K-1(config-if)#

From the above output we notice a few things. Immediately after associating the port channel with a FEX ID the FEX transitions to "Registered" then to "offline" while it begins to initialize. Next, we see the errors in the logs because I have not yet added the uplinks to N5K-2 to the VPC. Finally, we see the FEX come online. It can take a couple minutes for this process. Additionally, since fabric extenders are not managed independently, they download their image directly from the 5K. If the FEX needs to sync the image we will see the FEX transition to the "Image Download" state before proceeding to "online." The image download can take up to 15-ish minutes. Now we duplicate this config on the peer 5K:


N5K-2(config)#
N5K-2(config)# feature fex
N5K-2(config)# int e1/7-8
N5K-2(config-if-range)# channel-g 101 mod on
N5K-2(config-if-range)# int p101
N5K-2(config-if)# swi mod fex
N5K-2(config-if)# fex asso 101
N5K-2(config-if)# vpc 101
N5K-2(config)#
2012 Jan 16 06:10:25 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online
2012 Jan 16 06:10:25 N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line
N5K-2(config-if)# s2012 Jan 16 06:10:30 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online
N5K-2(config-if)# sh fex
  FEX         FEX           FEX                       FEX              
Number    Description      State            Model            Serial    
------------------------------------------------------------------------
101        FEX0101             Connected     N2K-C2248TP-1GE   SSI14280VS8
---       --------            Discovered     N2K-C2248TP-1GE   SSI14280VXT
N5K-2(config-if)#

I'll skip showing the config output here, but next we perform the same configurations for FEX 102 and perform final verification of the FEXes:


N5K-1(config-if)# sh fex
  FEX         FEX           FEX                       FEX              
Number    Description      State            Model            Serial    
------------------------------------------------------------------------
101        FEX0101                Online     N2K-C2248TP-1GE   SSI14280VS8
102        FEX0102                Online     N2K-C2248TP-1GE   SSI14280VXT
N5K-1(config-if)#

Similar to a 3750 switch stack, the Nexus shell creates a tiered interface structure when fabric extenders are added. The first tier is the fabric extender ID, the second tier is "1," and the third tier is the port number. Here's a sample "sh int status" command:


N5K-1(config-if)#
N5K-1(config-if)# sh int status fex 101

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
Eth101/1/1    --                 connected 1         full    1000    --        
Eth101/1/2    --                 connected 1         full    1000    --        
Eth101/1/3    --                 notconnec 1         auto    auto    --        
Eth101/1/4    --                 notconnec 1         auto    auto    --        
Eth101/1/5    --                 notconnec 1         auto    auto    --        
Eth101/1/6    --                 notconnec 1         auto    auto    --        
Eth101/1/7    --                 notconnec 1         auto    auto    --        
Eth101/1/8    --                 notconnec 1         auto    auto    --        
Eth101/1/9    --                 notconnec 1         auto    auto    --        
Eth101/1/10   --                 notconnec 1         auto    auto    --    


5. Configure host ports

Finally we're ready to configure a dual-homed host. Before that there's one final important note. Because the FEXes are full-mesh to the 5Ks, there will be duplicate ports to consider. In other words both 5K-1 and 5K-2 have an e101/1/1 interface and it is important to keep these configs identical on both switches! To further drive this point home, let's consider a host with two NICs. NIC1 will connect to FEX 101 and NIC2 will connect to FEX 102. We want these NICs to be bundled in an etherchannel. Even though there are only two physical connections (NIC1 & NIC2), we have four ports to configure on the Nexus 5Ks! e101/1/1 on 5K-1, e101/1/1 on 5K-2, e102/1/1 on 5K-1, & e102/1/1 on 5K-2. At this point the configuration is a standard etherchannel configuration:


N5K-1(config-if)#
N5K-1(config-if)# int e101/1/1
N5K-1(config-if)# channel-group 10 mode active
N5K-1(config-if)#

N5K-2(config-if)#
N5K-2(config-if)# int e102/1/1
N5K-2(config-if)# channel-group 10 mode active
N5K-2(config-if)#
N5K-2(config-if)# int e102/1/10-11
N5K-2(config-if)# channel-group 20 mode active
N5K-2(config-if)# 

Aucun commentaire :

Enregistrer un commentaire