Analyzing Cisco Command and Application Output to Isolate Problems Occurring at the Network Layer

This section describes several important Cisco IOS commands that display information that is useful for isolating network layer problems (see Table 9-3). Following that, a scenario-style example demonstrates how you can use the commands listed in Table 9-3 to help isolate a realistic network layer problem.

The first few commands listed in Table 9-3 are marked as general commands because they might reveal issues related to many aspects of configuration at the network layer (addressing, routing, optimization, symmetry, and so on). The next few commands concentrate on the ARP and IP routing table's contents and operations (through show and debug) and IP interfaces. The list of BGP show and debug commands makes a useful set for isolating problems that might spring from BGP configuration errors. IP traffic statistics, Internet Control Message Protocol (ICMP) messages, IP packet handling, and contents of active access lists are the focus of the last set of commands presented in Table 9-3.

Table 9-3 Cisco Commands to Isolate Network Layer Problems

Focus

Command

Description

General

ping {host / ip-address}

Sends an echo request packet to an address and then waits for a reply. The {host / ip-address} variable is the IP alias or IP address of the target system.

trace [destination]

Runs a trace. Displays the list of IP nodes between the local device and the destination IP address (IP host).

[no] debug ?

Displays a list of options for enabling or disabling debugging events on a device.

show running-config

Displays the current configuration of the Cisco device. (You might call it the currently running configuration file.)

Table 9-3 Cisco Commands to Isolate Network Layer Problems (Continued)

Focus

Command

Description

ARP

show ip arp

Displays all ARP table entries in cache.

debug arp

Displays information about ARP transactions.

Routing table (IP)

show ip route

Displays the current content of the routing table.

debug ip routing

Displays information on routing table updates and route-cache updates.

IP interface

show ip interface [brief]

Displays the list of local router's interfaces, their IP addresses, their physical and logical status, and so on. Adding the brief keyword displays a summary table showing all interfaces, their IP addresses (even those that are unassigned), and status.

You can inspect one particular interface at a time by typing the interface ID after this command. For example, show ip interface Ethernet 0

BGP (exterior routing)

show ip bgp

Displays the content of the BGP table (entries).

show ip bgp summary

Displays the list and status of all BGP connections (peers).

show ip bgp neighbors

Displays information about the TCP and BGP connections to neighbors.

show ip bgp flap-statistics

Displays information about BGP route flapping.

debug ip bgp [dampening | events | keepalives | updates]

Displays information that is related to BGP processing.

IP traffic

show ip traffic

Displays statistics about IP traffic, such as format errors, bad hops, encapsulation failures, unknown routes, and probe proxy requests.

debug ip icmp

Displays information about ICMP transactions.

debug ip packet

Displays general IP debugging information and IP Security Option (IPSO) security transactions.

IP access lists

show ip access-lists

[access-list-number / access-list-name]

Displays the content of all IP access lists or a specified numbered or named IP access list.

Now imagine that you are a network engineer at a corporation with offices in Washington, Baltimore, and Columbia. The networking devices at each location are named after the city in which they reside. You have console access to the router in Washington, which gives you IP connectivity to all devices in your network (see Figure 9-1).

Figure 9-1 Partial Network Diagram Showing the Connections Among Columbia (Access), Baltimore

(Distribution), and Washington (Core) and Their Dual Connection to the Corporate Core (Lenexa and Elmhurst)

Figure 9-1 Partial Network Diagram Showing the Connections Among Columbia (Access), Baltimore

(Distribution), and Washington (Core) and Their Dual Connection to the Corporate Core (Lenexa and Elmhurst)

You have determined that even though the prefix 198.133.219.0/24 (a network associated to Cisco Systems) is supposed to be in the routing tables of all the devices you manage, it is not present in any of them. To help isolate the problem, you first review the status of the current routing protocols on the router named Washington using the command show ip protocols. (See the output displayed in Example 9-2.) The output reveals that process EIGRP 202 is running and is redistributing BGP 21, which is also an active process on the Washington router and has neighbor relationships with 172.27.227.7 and 172.28.228.8.

Example 9-2 Output of the Cisco IOS show ip protocols Command

Washington>show ip protocol

Routing Protocol is "eigrp 202"

Redistributing: static, eigrp

202, bgp 21

Routing Protocol is "bgp 21"

Routing Information Sources:

Gateway Distance

Last Update

(this router) 200

4d15h

172.28.128.8 20

4d15h

172.27.227.7 20

4d15h

Distance: external 20 internal

200 local 200

Washington>

The route to Cisco's 198.133.219.0 can only be learned from Lenexa or Elmhurst through BGP. You decide to connect to Lenexa, a corporate core device, at 172.27.227.7 (one of Washington's BGP peers from Example 9-2) and check for the presence of the 198.133.219.0 prefix using the show ip route command (see Example 9-3). Lenexa's IP routing table (shown in Example 9-3) includes the 198.133.219.0 prefix.

Example 9-3 Output of the Cisco IOS show ip route Command

Lenexa>show ip route

172.17.

0.0/30

is subnetted, 1 subnets

C

172.

17.12.0

is directly connected,

FastEthernet0/13

B

172.22.

0.0/16

[20/0] via 172.27.227.2,

00:06:04

B

172.26.

0.0/16

[20/0] via 172.27.227.6,

4d22h

172.27.

0.0/27

is subnetted, 1 subnets

C

172.

27.227

0 is directly connected,

Vlan27

172.28.

0.0/28

is subnetted, 1 subnets

C

172.

28.128

0 is directly connected,

Vlan28

B

198.133

.219.0/24 [20/0] via 172.17.12.

1, 4d16h

B

213.173

.185.0/24 [20/0] via 172.17.12.

1, 4d16h

B

216.239

.33.0/24 [20/0] via 172.17.12.1

, 4d16h

S*

0.0.0.0/0 [1/0] via 172.17.12.1

Lenexa>

After you verify that the 198.133.219.0/24 route is on at least one of the corporate core devices (Lenexa), you can return to the console on Washington and examine its BGP routing table using the show ip bgp command (see Example 9-4). You discover that the 198.133.219.0/24 prefix is not in Washington's BGP table.

Example 9-4 Output of the Cisco IOS show ip bgp Command

Washington>show ip bgp

BGP table version

is 27, local router

ID is 172

.22.129.1

Status

codes: s suppressed

d damped,

h history

, * valid, > best, i

- internal

Origin

codes: i -

IGP, e -

EGP, ? - incomplete

Network

Next Hop

Metric

LocPrf Weight

Path

*> 172

21.0.0

172.28

128.1

0

77

11

i

*> 172

22.0.0

0.0.0.0

32768

i

s> 172

22.121.0/26

0.0.0.0

3850240

32768

i

s> 172

22.122.0/26

0.0.0.0

3850240

32768

i

s> 172

22.123.0/26

0.0.0.0

3850240

32768

i

s> 172

22.129.0/26

0.0.0.0

0

32768

i

* 172

23.0.0

172.27

227.3

0

77

31

i

*>

172.28

128.3

0

77

31

i

* 172

24.0.0

172.27

227.4

0

77

41

i

*>

172.28

128.4

0

77

41

i

* 172

25.0.0

172.27

227.5

0

77

51

i

*>

172.28

128.5

0

77

51

i

* 172

26.0.0

172.27

227.6

0

77

61

i

*>

172.28

128.6

0

77

61

i

Washington>

Having observed that the 198.133.219.0/24 prefix (BGP) is present in the Lenexa router (which is a peer of Washington) but absent from the Washington router, you decide to re-examine the status of the routing protocols on the Washington router using the show ip protocols command (see Example 9-5). You notice that a distribute list named CIT is applied to inbound routes from the BGP neighbors of Washington. You realize that you were not observant enough the first time because you did not notice this filter.

Example 9-5 Output of the Cisco IOS Command show ip protocols Showing a Distribute List Applied to the Inbound Updates Received from BGP Peers

Washington>show ip protocols

Routing Protocol is "eigrp 202"

Example 9-5 Output of the Cisco IOS Command show ip protocols Showing a Distribute List Applied to the Inbound Updates Received from BGP Peers (Continued)

Neighbor(s):

Address FiltIn FiltOut DistIn DistOut Weight RouteMap

Neighbor(s):

Address FiltIn FiltOut DistIn DistOut Weight RouteMap

172.27.227.7

CIT

172.28.128.8

CIT

Maximum path: 1

Routing for Networks:

Routing Information Sources:

Gateway Distance

Last Update

(this router) 200

4d15h

172.28.128.8 20

4d15h

172.27.227.7 20

4d15h

Distance: external 20 internal

200 local 200

Washington>

You then use the Cisco IOS show access-lists command to review the content of the access list named CIT (see Example 9-6). You know that networks (prefixes) that are not explicitly permitted by an access list are implicitly denied. You see that the access list named CIT does not permit the prefix 198.133.219.0/24 (Cisco Systems network). Therefore, the issue has been isolated to the missing network address in this access list.

Example 9-6 Output of the Cisco IOS show access-lists Command

Washington>show access-lists

Standard IP access

list 21

permit 172.28.

128.7

permit 172.28.

128.8

permit 172.27.

227.7

permit 172.27.

227.8

permit 172.22.

0.0, wildcard

bits 0

.0.255

255

Standard IP access

list CIT

permit 172.21.

0.0, wildcard

bits 0

.0.255

255

(5

matches)

check=200

permit 172.23.

0.0, wildcard

bits 0

.0.255

255

(8

matches)

check=192

permit 172.24.

0.0, wildcard

bits 0

.0.255

255

(12

matches)

check=180

permit 172.25.

0.0, wildcard

bits 0

.0.255

255

(10

matches)

check=170

permit 172.26.

0.0, wildcard

bits 0

.0.255

255

(10

matches)

check=160

Extended IP access

list dhcp_glean_acl

(per-user)

permit udp any

eq bootpc host 255.

255.255.255

eq

bootps

Washington>

The following example demonstrates how system log messages can help isolate network layer problems. While still in charge of the network displayed in Figure 9-1, assume that one day Network Operations calls you to report that it can't ping or Telnet into the access switch named Columbia_SW from Washington. Network Operations does not recall changing any of the configurations. Observe Figure 9-2, which focuses on the local devices of the network examined in the previous example.

Figure 9-2 Network Diagram Displaying Columbia Access Router, ColumbiaSW Access Switch, and the End Devices

Figure 9-2 Network Diagram Displaying Columbia Access Router, ColumbiaSW Access Switch, and the End Devices

To help isolate the problem, you decide to review the current interface status on Columbia using the Cisco IOS show ip interface brief command. You see that the link providing connectivity to Columbia_SW—namely interface FastEthernet 0/0—is up. (See the top portion of Example 9-7.) You then check for the Columbia router's CDP neighbors with the show cdp neighbor detail command. (See the bottom portion of Example 9-7.) You observe that CDP packets are being received from Columbia_SW; nevertheless, you notice another fact that you suspect is the cause of the current issues and decide to show it to Network Operations.

Example 9-7 Output of the show ip interfaces brief and show cdp neighbors detail Cisco IOS Commands

Columbia>show ip interface brief

Interface IP-Address

OK?

Method

Status

Protocol

Virtual-Accessl unassigned

YES

unset

up

up

FastEthernet0/0 unassigned

YES

manual

up

up

FastEthernet0/0.1 172.22.121

.1

YES

manual

up

up

FastEthernet0/0.2 172.22.122

.1

YES

manual

up

up

FastEthernet0/0.3 172.22.123

.1

YES

manual

up

up

Serial1/0 172.22.127

.1

YES

manual

up

up

Serial1/1 172.22.127

.129

YES

manual

up

up

Columbia>show cdp neighbors detail

Device ID: Columbia_SW

Entry address(es):

IP address: 172.22.121.1

Platform: cisco WS-C2950T-24, Capabilities: Switch IGMP

Interface: FastEthernet0/0, Port ID

(outgoing

port):

FastEthernet0/1

Columbia>

You call back Network Operations in Columbia and ask whether it has seen error messages on the console of Columbia or Columbia_SW. Network Operations tells you that it hasn't seen any. You ask it to input the Cisco IOS command show logging on Columbia to help isolate the issue. (You enter the command on Columbia as well so that you can discuss those results.) The results are shown in Example 9-8. You ask Network Operations to look at the output and see if console logging is reported as enabled or disabled. Your default practice is to have console logging enabled because console messages can provide useful information. Obviously, someone has disabled console logging (the first deviation from the baseline). You then suggest that Network Operations scan the output for reoccurring messages. Network Operations finally notices that there is a message reporting a duplicate IP address 172.22.121.1 on FastEthernet 0/0.1. This is indeed a network layer problem. You know from the baseline that Columbia should have an IP address 172.22.121.1 on FastEthernet 0/0.1, and Columbia_SW should use 172.22.121.2. If logging had been enabled, the isolation problem would have been trivial and noticed much earlier.

Example 9-8 Output of the show logging Cisco IOS Command Columbia>show logging

Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns) Console logging: disabled continues

Example 9-8 Output of the show logging Cisco IOS Command (Continued) Log Buffer (65536 bytes):

Dec 17 15:33:29: %IP-4-DUPADDR: Duplicate address 172.22.121.1 on FastEthernet0/0.1, sourced by 000a.8a44.de40

Columbia>

0 0

Post a comment