About the Authors

Barry Raveendran Greene is a Senior Consultant in the Internet Architectures Group of Consulting Engineering, Office of the CTO, Cisco Systems. Cisco's CTO Consulting group assist ISPs throughout the world to scale, grow, and expand their networks. The assistance is delivered through consulting, developing new features, working new standards (IETF and other groups), and pushing forward Best Common Practices (BCPs) to the Internet community. Barry's current topics of interests are ISP Operations...

Access

This is the edge of the ISP network facing the customer. The access layer can be made up of all kinds of devices, from a simple router supporting PSTN modem banks to cable-aggregation devices supporting broadband customers. For this reason, ISPs tend to subdivide their access layers into more manageable units. Larger businesses often have different business units running different access services, and it is useful to give these different units management access to the devices they are...

Access Control Lists Turbo ACLs

Turbo ACLs use a technique that takes a standard or extended ACL, creates a set of data tables, and compiles them for runtime processing. For this reason, Turbo ACLs also are referred to as compiled ACLs. Turbo ACLs do not change the first match wins characteristic of all ACLs. Instead, they reduce the number of CPU operations to find a match, allowing for larger ACLs to be used without an increase in packet latency. This provides ISPs with a tool to allow large ACLs without a significant...

Access List Types

Extended 48-bit MAC address access list IP standard access list (expanded range) IP extended access list (expanded range) Enables IP access-list compilation (new from 12.0(6)S) Simple rate limit-specific access list Specifies a dynamic list of permits or denies KA9Q NOS-compatible IP over IP tunneling A single source host (equivalent to a.b.c.d 255.255.255.255) Log matches against this entry, including input interface Matches packets with given precedence value Matches packets with given TOS...

Access Lists on the VTY Ports

It is important to secure the VTY ports used for Telnet access with a standard ACL. By default, there are no access controls on any of the VTY ports. If this is left this way and a password is applied l to the VTY port, the router will be wide open to anyone who attempts a brute-force crack against the password. The following configuration with access-list 3 is typical of a better approach aaa authentication login Cisco-Lab local username Ciscol password 7 11041811051B13 access-list 3 permit...

Address Space for Customers

Customers are assigned address space by the ISP according to the policies set out by the RIR of which the ISP is a member. As an approximate guide, ISPs are expected to follow the same procedures with their customers as they have followed with the RIR in obtaining address space. So, when the ISP makes first technical contact with the customer, an assessment of address space requirements is made according to need. Need should not be confused with want. Staff members from many organizations who...

Appendix B Cutand Paste Templates

The following are some cut-and-paste templates that you can modify to configure your routers. Make sure that you change any sample IP addresses or AS numbers used in the templates to match your own addressing Do not use the addresses in the examples because they are invalid. As described in the main text, it is considered good practice to set up a configuration template for each class of router running in the network. Use these templates, taken from running configurations in ISP backbones...

BGP fastexternalfallover

By default, if the physical connection to the eBGP neighbor goes down, the peering relationship is reset immediately. By adding the no bgp fast-external-fallover configuration, the peering is held open for the duration of the BGP keepalive timer. This configuration is desirable, if not essential, in the case of long-distance peering links or unreliable or long-latency connections to other autonomous systems, and when ISPs prefer stability over convergence speed in large networks. router bgp 109...

BGP Filter Processing Order

When constructing BGP filters, most ISPs use prefix lists or distribute lists, AS path filters, and route maps to implement their filtering policies. It is important to realize that these lists are processed in a particular order newer engineers sometimes are unaware of the standard IOS Software sequence. In the following, example the IOS Software configuration generator (NVGEN) has printed the configuration into the previous order 0.1 prefix-list rfc1918-out out Implementation order is...

BGP Flap Damping Configuration

Recommended route flap-damping parameters for use by ISPs were composed into a document by the RIPE Routing Working Group and are available at www.ripe.net docs ripe-229.html. These values are used by many European and U.S. ISPs, and they are based on the operational experienced gained in the industry. The configuration examples are reproduced here for convenience the values have been updated to include recent changes in the locations of the root nameservers. The current address list is...

BGP Flap Statistics

It is possible to monitor the flaps of all the paths that are flapping. The statistics will be lost when the route is not suppressed and stable for at least one half-life time. The display looks like the following cerdiwen sh ip bgp neighbors 171.69.232.56 flap-statistics BGP table version is 18, local router ID is 172.19.82.53 Status codes s suppressed, d damped, h history, * valid, > best, i - Origin codes i - IGP, e - EGP, - incomplete Network From Flaps Duration Sup-time Path *> 5.0.0.0...

BGP Peer Groups

The second essential feature is the BGP peer group. This groups the BGP peers with the same outbound policy into one group. The normal situation without using peer groups is for the router to calculate the update to be sent to each neighbor individually. For a low number of peers, there is probably not much impact on the router CPU, but as the number of neighbors increases, so does the burden on the CPU for example, 50 neighbors means that 50 updates have to be computed and sent individually....

BGP Route Flap Damping

Route flap damping (introduced in Cisco IOS Software at Release 11.0) is a mechanism for minimizing the instability caused by route flapping. A route flap occurs when a BGP network prefix is withdrawn and reannounced specifically, this happens when a BGP speaker hears a WITHDRAW followed by an UPDATE for a prefix. (A peering with an eBGP neighbor being reset does not count as a flap.) Whenever a network goes down, the rest of the Internet is told about it. Hence, BGP propagates this state...

BGP Update Source

In the following example, the iBGP mesh is built using the loopback interface on each router. The loopback doesn't ever disappear, which results in a more stable iBGP, even if the underlying physical connectivity is less than reliable. ip address 215.17.1.34 255.255.255.255

Black Hole Routing as a Packet Filter Forwarding to NullO

Another way of implementing destination-based packet filtering on a router is to create a specific list of static host routes and point them to the pseudo-interface NullO. This technique commonly is referred to as black-hole routing. Null0 is a pseudo-interface, which functions similarly to the null devices available on most operating systems. This interface is always up and can never forward or receive traffic. Although Null0 is a pseudo-interface, within CEF it is not a valid interface....

Car

Active defenses are tools, techniques, or procedures executed during attacks. They are used to limit or block the attack in progress. In many instances, these tools will have an effect on other Internet applications and services, yet the trade-off is between no Internet services and limited disruption. It is highly recommended that the ISP document and train staff on the use of these tools. That way, the ISP's NOC can quickly respond to an attack in progress. Cisco IOS Software has security...

Case Study

The following case study implements the techniques described previously for an ISP that recently needed to multihome to the Internet. The scenario is quite simple, but it covers a situation that causes many newcomers to BGP and the Internet quite a lot of trouble. The network in Figure 5-31 shows the network layout. AS 5400 is based in Europe, and AS 2516 is based in Japan this in itself poses a challenge because the multihoming is between entities that are quite literally on opposite sides of...

Caution

Do not remove the enable password as in the previous example if the boot ROMs or boot image of the router does not support the enable secret configuration. The use of enable secret is supported in IOS Software Release 11.0 and later. With an older boot ROM and no enable password, it is possible to gain access to the router without supplying any password if the router ends up running the boot image because of some network problem or malfunction. A network's first line of defense is the routers...

Choosing an IGP

A general discussion on the appropriate technical choice of IGPs for an ISP backbone is currently beyond the scope of this book. Many good comparisons have been done describing the pros and cons of the different IGPs. A commonly quoted example is the presentation by Dave Katz at the June 2000 NANOG comparing IS-IS and OSPF. The choice of IGP generally seems to be made on the basis of experience because technically there is little way to choose among the three for most practical purposes. Those...

Cisco IOS Software release designations definedSoftware Lifecycle Definitions

Software naming conventions for Cisco IOS Cisco IOS reference guide Cisco IOS roadmap Cisco Resource Manager Private I http www.opensystems.com index.asp Crystal Reports Netforensics http www.netforensics.com NTP RFCs RFC 1128, RFC 1129, RFC 1165, and RFC 1305, all available at http www.ietf.org rfc

CLI String Search

After a considerable number of requests from ISPs, a UNIX qrep-like function (pattern search) has been introduced as a new feature in IOS Software from releases 11.1CC and 12.0. It allows operators to search for common expressions in configuration and other terminal output. Again, only salient points are covered here because the IOS Software documentation now gives more detailed information at 20t1 cliparse.ht . The function is invoked by using a vertical bar , like the UNIX pipe command begin...

Committed Access Rate CAR

Configuring Committed Access Rate c f qcprt1 qcfcar.ht RFC 1812. Requirements for IP Version 4 Routers. F. Baker (ed). June 1995. (Status Proposed standard.) Also see the update, RFC 2644. RFC 2196 FYI8. Site Security Handbook. B. Fraser. September 1997. (Obsoletes RFC 1244) (Status Informational.) One of the most useful starting places for Internet security. RFC 2827 BCP 38. Network Ingress Filtering Defeating Denial-of-Service Attacks Which Employ IP Source Address Spoofing. P. Ferguson and...

Comparing Router IDs

As part of the standard path-selection process in IOS Software, the router does not switch between two eBGP paths based solely upon the router ID. Consider a situation in which a router hears two announcements of a particular prefix from two different neighbors. If the path-selection process determines that all attributes are identical apart from the router ID, it currently takes the path with the oldest entry in the routing table, not the entry with the lowest router ID. This choice was made...

Conclusion

This section gave an example of how to work out an addressing scheme for a developing ISP network. This is intended to help the growing ISP business work out how to apply addressing to its network and how to allocate assigned address space to its infrastructure. Indeed, following these processes should aid in the application process for address space from the RIRs. One of the hardest questions that a vendor is posed at the time of an RFP is which IGP the new ISP network should choose. As...

Conditional Advertisement Example

Consider the example depicted in Figure 3-8. This shows a dual-homed enterprise network (AS 300) that has received address space from its two upstream ISPs. It announces the 215.10.0.0 22 prefix to ISP 1 (AS 100) and the 202.9.64 23 prefix to ISP 2 (AS 200). These networks are part of the respective upstream ISPs address blocks, so all that the Internet sees are the two aggregates as originated by ISP 1 and ISP 2. This is the steady state situation. Figure 3-8. BGP Conditional Advertisement...

Creating Your Own Net Police Filter

ISPs that want to create their own net police filters are strongly encouraged to do the following Consider the impact Shutting the door doesn't make the storm go away it shuts out everything, good and bad. Maintain an accurate list Consult each of the RIR's published CIDR blocks for the default allocations. Ensure that you have an up-to-date list, and create a process for validating your filter with future updates of the RIR's list. Consult with colleagues Consult with your peers about the...

Customer networks Loopback Interfaces

The loopback interface on the router is always the first consideration on an ISP network. It is a helpful general-purpose feature used for many things, including iBGP peering and source address for packets originating from the router (useful for authentication or filtering). The early chapters of this book made many references to the loopback interface, and a whole section was devoted to the benefits of using its address as the source of all IP packets originating from the router. By the end of...

Customer Router Filters

As a service to their customers, all ISPs should be supplying sample filters for routers that are used to connect permanently connected networks to the Internet. If the customers are using routers that are incapable of filtering, to quote an ISP overheard recently, that device should be replaced with a real router. It is an unfortunate fact today that too many people equate security with firewalls and completely fail to remember that a router is a very sophisticated first-line security device...

Designing a Test

From the description of the previous test scenario, it should be quite clear what components make up an ISP's test lab. Some ISPs build a replica of one of their PoPs others simply have a few of the major devices used on their backbone connected in a simple network. Preference usually is given to the PoP replica design because it becomes very simple to replicate problems that occur on the live network. Several ISPs even have this test lab as part of their backbones the lab won't take an active...

Displaying BGP Policy Accounting Status

To inspect which prefix is assigned which bucket and which communities, do the following 196.240.5.0 24, version 21, cached adjacency to POS7 2 packets, 0 bytes, traffic_index 4 Look Here via 14.1.1.1, 0 dependencies, recursive next hop 14.1.1.1, POS7 2 via 14.1.1.0 30 valid cached adjacency BGP routing table entry for 196.240.5.0 24, version 2 Paths (1 available, best 1) Not advertised to any peer 100 14.1.1.1 from 14.1.1.1 (32.32.32.32) Origin IGP, metric 0, localpref 100, valid, external,...

Distribution

The distribution layer is one step removed from the core and gets its name from its function of acting as a distribution layer between the core routers and the access part of the network. Indeed, many small- to medium-size ISPs don't have any distribution layer they simply connect the access part of the network to the core. It all depends on the size of the PoP. The distribution layer can be made up of two or more routers quite often there could be considerably more. ISPs conscious of providing...

EBGP Multihop

Diversely Multihomed Stub Network Figure 5-21. Diversely Multihomed Stub Network eBGP multihop is an eBGP peering between the loopback interfaces (or other interface not on the demarcation zone between the two networks) of routers in the two networks. The configuration could be something like the following Router A router bgp 65534 neighbor 1.1.1.1 remote-as 100 neighbor 1.1.1.1 ebgp-multihop 5 ip route 1.1.1.1 255.255.255.255 serial 1 0 ip route 1.1.1.1 255.255.255.255 serial 1 1...

Editing Keys

Several keys are very useful as shortcuts for editing the IOS Software configuration. Although these are covered in detail in the IOS Software release 12.0 documentation set, it is useful to point out those used most commonly, shown in Table 1-2. Completes the command being typed in. This saves typing effort and is especially useful when the operator is still learning the IOS Software command set. Lists the available commands starting with the characters entered so far. Allows the operator to...

Effects of CIDRization

One of the most critical issues that could threaten the stability of the Internet is the size of the global Internet Route Table. The table's growth influences scalability, increases operational capital cost, and has posed a security risk to the Internet. For a variety of reasons, ISPs throughout the world have injected all sorts of networks into the Internet, ranging from 8s (old Class A's) to 32s (host routes). The result is rapid growth of the Internet Route Table. Classless interdomain...

Example 21 Sample Output from Displaying Flow Information on a Net FlowEnabled Router

IP packet size distribution (410772243 total packets) 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .168 .384 .102 .160 .107 .019 .005 .003 .001 .001 .000 .000 .000 .003 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .001 .000 .035 .000 .003 .000 .000 .000 .000 .000 .000 IP Flow Switching Cache, 4456704 bytes 15074 active, 50462 inactive, 125120769 added 369493980 ager polls, 0 flow alloc failures last clearing of statistics 4d05h Total Flows 605 3494 4104 845158 87119 59...

Example 41 Information That Can Be Gained from CDP

Device ID Excalabur Entry address(es) IP address 4.1.2.1 Platform cisco RSP2, Capabilities Interface FastEthernet1 1, Port ID Holdtime 154 sec Cisco Internetwork Operating System Software IOS (tm) RSP Software (RSP-K3PV-M), Version 12.0(9.5)S, EARLY DEPLOYMENT MAINTENANCE INTERIM SOFTWARE Copyright (c) 1986-2000 by cisco Systems, Inc. Compiled Fri 03-Mar-00 19 28 by htseng CDP can be disabled using this global command If CDP is required on an ISP's network, it is possible to leave CDP running...

Example 45 Multihomed uRPF

Interface serial 1 0 1 description Link to Acme Computer's Router C ip address 192.168.3.2 255.255.255.252 ip verify unicast reverse-path no ip redirects no ip directed-broadcast no ip proxy-arp ip route-cache distributed neighbor 192.168.10.3 remote-as 65000 neighbor 192.168.10.3 description Multihomed Customer - Acme Computers neighbor 192.168.10.3 update-source Loopback0 neighbor 192.168.10.3 send-community neighbor 192.168.10.3 soft-reconfiguration inbound neighbor 192.168.10.3 route-map...

Example B1 General System Template

No service finger replaced with ip finger from 12.0 service timestamps debug datetime localtime show-timezone msec service timestamps log datetime localtime show-timezone msec interface loopback 0 description Loopback interface for router XY Enable logging with two loghosts using facility local4 - these commands are default from 12.0 - best included anyway Enable Cisco Express Forwarding

Example B3 General Security Template

Service password-encryption enable secret < removed> no enable password aaa authentication login default tacacs+ enable aaa authentication enable default tacacs+ enable And set up the TACACS+ authentication - two servers ip tacacs source-interface loopback 0 tacacs-server host z.z.z.A tacacs-server host z.z.z.B tacacs-server key < removed> - need to run crypto key generate rsa before applying this template Protect the console ports - list NOC and other permitted addresses in access-list...

Example B7 Prefix List to Deny RFC 1918 and Martian Networks

Ip Networks which shouldn't be announced ip prefix-list rfc1918-sua deny 0.0.0.0 8 le 32 ip prefix-list rfc1918-sua deny 10.0.0.0 8 le 32 ip prefix-list rfc1918-sua deny 127.0.0.0 8 le 32 ip prefix-list rfc1918-sua deny 169.254.0.0 16 le 32 ip prefix-list rfc1918-sua deny 172.16.0.0 12 le 32 ip prefix-list rfc1918-sua deny 192.0.2.0 24 le 32 ip prefix-list rfc1918-sua deny 192.168.0.0 16 le 32 ip prefix-list rfc1918-sua deny 224.0.0.0 3 le 32 ip prefix-list rfc1918-sua deny 0.0.0.0 0 ge 25

Example C2 Core Router Configuration Example

Version 12.0 service nagle no service pad service tcp-keepalives-in service timestamps debug datetime msec localtime show-timezone service timestamps log datetime msec localtime show-timezone boot system flash slot0 c7200-k4p-mz.120-10.S2 logging buffered 16384 debugging aaa new-model aaa authentication login default tacacs+ enable aaa authentication enable default tacacs+ enable aaa accounting exec default start-stop tacacs+ aaa accounting commands 15 default start-stop tacacs+ ip subnet-zero...

Example C3 Aggregation Router Configuration Example

Version 12.0 service nagle no service pad service tcp-keepalives-in service timestamps debug datetime msec localtime show-timezone service timestamps log datetime msec localtime show-timezone service password-encryption logging buffered 16384 debugging aaa new-model aaa authentication login default tacacs+ enable aaa authentication enable default tacacs+ enable aaa accounting exec default start-stop tacacs+ aaa accounting commands 15 default start-stop tacacs+ clock timezone GMT 0 ip...

Example C4 Service Router Configuration Example

Version 12.0 service nagle no service pad service tcp-keepalives-in service timestamps debug datetime msec localtime show-timezone service timestamps log datetime msec localtime show-timezone aaa authentication login default tacacs+ enable aaa authentication enable default tacacs+ enable aaa accounting exec default start-stop tacacs+ aaa accounting commands 15 default start-stop tacacs+ clock timezone GMT 0 ip subnet-zero no ip source-route no ip finger ip telnet source-interface Loopback0 ip...

Example C5 NOC Router Configuration Example

Service timestamps debug datetime msec localtime show-timezone service timestamps log datetime msec localtime show-timezone aaa authentication login default tacacs+ enable aaa authentication enable default tacacs+ enable aaa accounting exec default start-stop tacacs+ aaa accounting commands 15 default start-stop tacacs+ clock timezone GMT 0 ip subnet-zero no ip source-route no ip finger ip telnet source-interface Loopback0 ip tftp source-interface Loopback0 ip ftp source-interface Loopback0 ip...

Examples of the New CLI in

This section gives some examples of the revised CLI. They have been taken from real, live running configurations working in networks today. The first example shows how BGP supporting IPv6 is configured. This router is a part of the 6BONE, the IPv6 experimental backbone no bgp default ipv4-unicast allow more than IPv4 uni bgp log-neighbor-changes bgp dampening neighbor UPSTREAMS peer-group neighbor iBGP-peers peer-group neighbor 2001 200 0 1805 2 remote-as 2500 neighbor 3FFE 800 FFF9 0 0 9...

Figure 31 Dual Gateway LAN

Interface ethernet 0 0 description Server LAN ip address 169.223.10.1 255.255.255.0 standby 10 ip 169.223.10.254 interface ethernet 0 0 description Server LAN ip address 169.223.10.1 255.255.255.0 standby 10 ip 169.223.10.254 interface ethernet 0 0 description Service LAN ip address 169.223.10.2 255.255.255.0 standby 10 priority 150 standby 10 preempt standby 10 ip 169.223.10.254 The two routers have their LAN IP addresses conventionally defined in the preceding configuration. However, another...

Figure 415 Egress Packet Filtering on the Upstream Gateway Router

Allow source address 16S.21.0.0 16 < - - _ -- Allow source address 16S.21.0.0 16 < - - _ -- Ex. IP packeis win n unce adftfew nf 10.1.1.1 Awls btoctad, access-list 110 permit ip 165.21.0.0 0.0.255.255 any access-list 110 deny ip any any log interface serial 0 1 description Upstream Connection to ISP A ip access-group 110 out The last line of the access list determines whether there is any traffic with an invalid source address entering the Internet. If there are any matches, they will be...

Figure 422 uRPF Dropping Packets that Fail Verification

Fddi 2 0 fl 172.15.8,0 attached Fddi 2 0 0 216.SlO.e.a 172.ia.ee. Fddi 2 0 fl 172.15.8,0 attached Fddi 2 0 0 5. If an uRPF ACL is applied, the packet is processed through that feature ACL before final dropping. This ACL could be configured to overrule the uRPF check and pass the packet. 11 6. CEF table (FIB) lookup is carried out for packet forwarding, passing packets that match the FIB + adjacency check or dropping packets that are spoofed sources. For example, if a...

Figure 43 Command Auditing on the Router Through Cisco Secure and TACACS

Configuration control and audit of who has done what and when on the routers is the key objective for using AAA command accounting on an ISP's backbone. aaa authentication login default tacacs+ enable aaa authentication enable default tacacs+ enable aaa accounting command 15 start-stop tacacs+ aaa accounting exec start-stop tacacs+ ip tacacs source-interface Loopback0 tacacs-server host 215.17.1.2 tacacs-server host 215.17.34.10 tacacs-server key CKr3t

Figure 430 How Smurf Uses Amplifiers

Each host on that IP network will take the ICMP echo request and reply to it with an echo reply. This multiplies the inbound traffic by the number of hosts responding. On a multi-access broadcast network, potentially hundreds of machines could be replying to each packet, resulting in what is called an attack from a smurf amplifier network. The systems most commonly hit by these types of attacks are Internet Relay Chat (IRC) servers, specific Web sites, and their providers. Two parties are hurt...

Figure 48 Ingress Filtering

Egress filtering applies a filter for all traffic leaving an ISP's networks (see Figure 4-9). It is applied to information leaving the network to the Internet or customer networks. Be mindful that these terms are relative to the specific network's point of view. For example, ISP B's egress traffic is ISP A's ingress traffic. Ingress egress filters help protect an ISP's resources and its customers' networks, allows it to enforce policy, and minimizes the risk of being the network chosen by...

Figure 511 RIR Areas

It is hoped that this section will encourage ISPs to consider how to design a scalable addressing plan. Conservation and efficient utilization of address space often are seen as problematic and even undesirable by ISPs trying to minimize the number of prefixes carried around in their network. This is only an example of the considerations necessary when designing the addressing plan for an ISP network. It does not advise on how to go through the process of applying for address space from the...

Figure 529 Two Upstream ISPs

Again, the common solution to this problem is for both upstream ISPs to provide the full routing table to AS 109, as can be shown in the following router configuration examples router bgp 109 network 221.10.0.0 mask 255.255.224.0 neighbor 222.222.10.1 remote-as 107 neighbor 222.222.10.1 prefix-list rfc1918-deny in neighbor 222.222.10.1 prefix-list my-block out neighbor 222.222.10.1 route-map AS107-loadshare in ip prefix-list my-block permit 221.10.0.0 19 See Appendix B for the RFC1918 list ip...

Figure 533 Router B Upstream Link Load

Over time, this configuration will have to be monitored because traffic patterns do change as networks grow and the clientele on the network come and go. But as a case study for good multihoming practice, this is one of the best efforts seen on the Internet today. Unlike the claims from some of the potential upstream ISPs of AS 17660, all this has been achievable with minor router platforms and minimum memory requirements (the two 2600s each still have around 32 MB of memory spare, plenty of...

Figure 55 PoP Access Network

Most ISPs don't bother too much with the redundancy of the RAS. Several will provision two switches, but it is rare for them to provision two Ethernets out of the RAS. The top-end RAS devices have two Ethernets, so better redundancy can be provided. However, in most cases, ISPs tend to use low-end units and large numbers of these because failures are actually easier to deal with. For other types of access networks, there will be other designs. Cable and xDSL access require a much more...

Figure F3 Closing Off Access to Everyone Except the NOC Staff

Example F-3 ACLs with Telnet Access Closed to All but the NOC's Network Example F-3 ACLs with Telnet Access Closed to All but the NOC's Network aaa authentication login ISP local username Ciscol password 7 11041811051B13 access-list 3 permit 211.255.1.0 0.0.0.255 line vty 0 4 access-class 3 in exec-timeout 5 0 transport preferred none transport input telnet login authentication ISP history size 256

Further Reference on IOS Software Releases

Figures 1-1 and 1-2 provide a visual map of IOS Software releases up to 12.1 they also show how the different versions and trains interrelate. This has been and still is an often-asked question in the ISP arena and other marketplaces in which Cisco is present these visual roadmaps have been created to show the interrelation of the different IOS Software versions. The current up-to-date roadmap can be seen at Consult the following URLs on Cisco.com for more detailed and up-to-date information on...

IGP Adjacency Change Logging

Neighbor state logging should be enabled in each IGP. This means that it becomes easier to find out about neighbor states, reasons for state changes, and so on. For each IGP, the IGP subcommand log-adjacency-changes enables logging. (Some older versions of IOS Software require the IGP to be specified as well for example, ospf log-adjacency-changes.) By the time this book is published, the command will likely be enabled by default in IOS Software. If logging is enabled, log messages are sent to...

Internet Traffic and Network Engineering Tools

This section lists just some of the many Internet traffic and network engineering tools available. These are the tools that ISPs use to monitor their networks, help monitor traffic flows, and determine many business-critical functions such as peering policies, QoS, and so on. Stan Barber presented a talk at the February 1998 NANOG titled Monitoring Your Network with Freely Available Statistics Reporting Tools. The slides for this presentation are available at and are recommended reading for all...

IOS Software Regular Expressions

Here are some examples of regular expressions used for BGP peerings on ISP routers today. Refer to the documentation for more in-depth discussion and detailed examples. Matches all paths, including the local AS Matches all autonomous systems received from AS200 Matches all paths with AS200 as the origin Matches AS210 origin and received from AS200 only Matches all paths that have been through AS200 and AS210 link Matches at least one of AS200 (or multiple occurrences of one AS) (usually from...

ISP Addressing Plan

For reference purposes, the following addressing plan has been used for this ISP backbone. The address block is 220.144.128.0 19. 220.144.128.0 and upwards to be used for customer assignments .160 through .175 Host range for terminal server console port access Out-of-band management terminal server

ISP Server Considerations

Servers within the ISP or customer network need to be properly set up and secured before they are plugged into any LAN or even connected to the Internet. ISP servers should be connected behind a router with strong filters, never plugged into the core of the backbone, as has been mentioned previously. Using a router gives the ISP a chance to install strong filters to protect the servers. For example, if the server is a Web server, only port 80 should be visible to the outside world, and so on....

Keeping Software Upto Date

A common error made by many ISPs is relying on the server software that is distributed with their UNIX systems rather than keeping the versions current. Mail and DNS are two extremely high-profile services, and the developers and user community continuously are making enhancements and bug fixes. It's quite common for operations staff to subscribe the ISP to the CERT advisory mailing list here all advisories about known security problems with software on UNIX- and Windows-based operating systems...

Load Sharing

The second example refines the previous one so that both links to the two upstream ISPs can be used to carry traffic at all times. Figure 5-27 shows the scenario. The way to implement load sharing here is to start by subdividing the 19 into two 20s the 19 aggregate still is announced out of each link, but the announcement of a 20 on each link will ensure that traffic for that 20 will follow that path by preference. The configurations of the routers are given as follows router bgp 107 network...

Looking Glasses

A looking glass isn't really a UNIX tool as such it's more a facility that has been made available by a large number of ISPs and other organizations around the globe. It provides a web interface to users, allowing them to check conditions of the Internet routing table at that site. A complete list of looking glass sites can be found at http www.traceroute.org. The most well known and one of the first looking glasses was the one set up by Ed Kern at Digex. It can be found at http...

Multihomed Stub Network

If the network in the preceding example is multihomed on to their upstream by connecting to different routers in their upstream's network, BGP will have to be used. Some ISPs use an IGP for this function (we have seen RIP, EIGRP, and OSPF all being used), but this practice is strongly discouraged because it is a very serious potential source of misconfiguration and problems in the ISP backbone. The chances of having the customer's IGP leaking into the ISP's IGP are very great such...

Multiple Dual Homed Customers RFC 2270

RFC 2270-based multihoming is an extension of the previous two examples, describing how to scale a situation in which multiple customers are multihoming onto the ISP backbone. Figure 5-25 shows how multiple customer would multihome onto the backbone of the same upstream ISP. The diagram also shows that the same ASN is used this is not a drawing error, but it is genuinely all that is required. The configuration will be based on that used in the previous section, so AS 100 routers will be...

Multiple Upstream ISPs and IXP

The third example adds a little more complication to the network and is probably one of the more extreme cases found on the Internet today, as shown in Figure 5-30. This is an example of a well-connected ISP, present at the local exchange point, with a few private peers, some regional ISP connections providing backup transit, and a couple of upstream Tier 1 ISPs providing Internet transit. The configurations for this aren't too hard to generate, either. Following the previous examples, the...

Net Flow Data Export

The greatest benefits of NetFlow are found when its data is exported to collection systems and then are analyzed and processed. Cisco has adopted a broad approach to facilitate this activity. These include donations for freeware collection analysis software, Cisco's own commercial software, tools for others to create their own software, and partnerships with companies that make commercial-grade billing systems based on NetFlow export. To export the data, the following configuration commands are...

Network Design

Good attention needs to be paid to the design of the news-delivery network. For a network engineer, it should not be something that is simply left to the systems group to work out for themselves because the volume of material that can be delivered often can cause choke points in the backbone. Furthermore, paying close attention to the design will ensure that the customer's experience is a good one and will not add more fuel to the Internet is slow perception. Figure 5-9 shows an example of how...

No IP Unreachables

For a long time, Cisco routers had the configuration capability to turn off ICMP Unreachable response. This was done with the interface command no ip unreachables. Whether this is done is an operational decision of the ISP some do and some do not. The router requirements RFC (RFC1812) says that each device should respond with ICMP Unreachables, but when a network operator experiences an attack against a router, RFC niceties get left behind. What can be recommended or considered is that no ip...

Normal iBGP peer group For normal iBGP peers

iBGP client peer group For reflection peers on a route reflector. eBGP full routes For peers to receive full Internet routes. eBGP customer routes For peers to receive routes from direct customers of the ISP only. Some members can be configured with default-origination to receive the default route as well as the customer routes. eBGP default routes For peers to receive the default route and possibly a few other routes.

Note

Routing loops occur when two or more routers select the best path back to the other. This causes packets to ping pong back and forth until the packet's TTL times out and drops the packet. BGP attributes such as Local Preference allow the ISP to control the flow of traffic between its network and other providers' networks. It does not allow them to effectively influence the best-path decisions of other networks. So, in one common form of asymmetrical routing, each ISP makes its own best-path...

NTP Architecture [2

In the NTP model, a number of primary reference sources, synchronized by wire, GPS, or radio to national standards, are connected to widely accessible resources, such as backbone gateways, and are operated as primary time servers. NTP provides a protocol to pass timekeeping information from these servers to other time servers from the Internet and to cross-check clocks and correct errors arising from equipment or propagation failures. Local-net hosts or gateways, acting as secondary time...

NTP Source Interface

NTP is the means of keeping the clocks on all the routers on the network synchronized to within a few milliseconds. If the loopback interface is used as the source interface between NTP speakers, it makes filtering and authentication somewhat easier to maintain. Most ISPs want to permit their customers to synchronize only with their time servers, not everyone else in the world. Look at the following configuration example access-list 5 permit 169.223.50.14 ntp authentication-key 1234 md5...

Nvram Tftpserver and FTPserver

The onboard router NVRAM is used to store the router's active configuration. Most ISPs keep an off-router copy of this configuration, too. In the unlikely event that the configuration is lost on the router, they can quickly recover the system with the off-router backup. There are several options for off-router backup of the running configuration Write configuration to a TFTP server using the write net command. (In IOS Software release 12.0 and more recent software, the write net command has...

One Time Password

One advantage of ensuring that the ISP's operations team uses TACACS+ to access to the network's infrastructure is that one-time password (OTP) techniques can be used to provide an additional layer of security. The common approach usually is to use a smart card in the possession of the individual being authenticated. That individual enters a personal identification number (PIN) into the smart card. The smart card returns a time-sensitive password that can be used as the authentication password...

Operational Considerations

Why design the world's best network when good operational practices have not been considered This might be such an innocent question, but it is surprising how many new ISPs completely forget about any operational practices for their networks. The best-designed network can work only as well as the operators who are running it. Likewise, good operational practices often can make up for a lot of deficiencies in the physical layout of networks. This section highlights some of the issues that need...

OTP Configuration Hints

OTP logins result in traffic to TACACS+ server, to the OTP service, back to the TACACS+ server, and then to the network device. Hence, the network device should be configured with a minimum 10-second TACACS+ timeout setting The default TACACS+ timeout setting of 1 second will cause OTP logins to fail a large percentage of the time. For complete and detailed information on OTP configuration with CiscoSecure ACS, check the online documentation at www.cisco.com. Look for the chapter titled Token...

Outof Band Circuits

The terminal server usually is connected to the management LAN in the PoP. But if access to the PoP is completely disconnected from the outside world because the ISP's equipment completely fails or all the telco circuits fail, the ISP still will need access to the equipment to aid in restoring the service. (For example, the two core routers might have crashed, causing all external access to be disconnected. Regaining access to these routers gives the NOC some chance of restoring connectivity to...

Outof Band ISDN

Some ISPs use ISDN as the means of accessing their out-of-band management system. ISDN is readily available in many countries, often at a cost not too dissimilar to PSTN, but without the need for an external modem attached to the router. Experience has shown that modems can become faulty, so having the enhanced reliability available through ISDN is an advantage for some. An added advantage is that ISDN supports 128 Kbps, making real-time upgrades of router or switch images that much more...

Phase 1Close Off Access to Everyone Outside the CIDR Block

The theme in Phase 1 is to get the ball rolling by limiting access to just those IP addresses inside the ISP's CIDR block. A standard ACL is created to permit Telnet only from the IP addresses in the CIDR block. This ACL is used with the VTY's access-class command to ensure that the source IP address of any Telnet packet coming to the VTY port matches the ACL. Why just the ISP's CIDR block First, it's an easy-to-implement technique for an ISP that has no filters. The ISP does not have to worry...

Point of View

39 90 91 92 93 94 95 96 97 93 99 00 01 The following is a list of sites collecting data on the Internet Route Table Tony Bates's Daily CIDR Report Philip Smith's Daily Routing Analysis http www.apnic.net stats bgp Geoff Huston's BGP Table Data 39 90 91 92 93 94 95 96 97 93 99 00 01 The second approach to encouraging ISPs to use proper aggregation was to install inbound prefix filters on route announcements. Sean Doran (smd clock.org) was one of the first to notice and act on the fact that the...

Primary and Backup Paths

The first example considered (shown in Figure 5-23) is one with two paths between the networks One path is used as the primary link, and the other path is used exclusively for backup. This situation is used commonly when the primary path has a high bandwidth and the backup path is of low bandwidth or poor latency and is sufficient only when the main link has failed. Figure 5-23. Primary and Backup Paths to the Same ISP Figure 5-23. Primary and Backup Paths to the Same ISP The primary path is...

PSA ACLs in the Cisco 12000 Engine 2 Line Card

The Cisco 12000's Engine 2 line card took the approach of using the spare capacity in the PSA forwarding ASIC to apply ACLs. Specific microcode needs to be loaded when the hardware-based ACL is applied to the PSA. The result is significant PPS performance, with ACL depth determined to be acceptable for most ISP operations (around 128 lines of ACE with 448-line capability). Although there is a PPS advantage with this implementation, there are also limitations. If these limitations are exceeded,...

Pushing Out a Prefix List ORF

The command to push out a prefix-list ORF and receive route refreshes from a neighbor is given here clear ip bgp x.x.x.x in prefix-filter When the inbound prefix list changes (or is removed), this command can be used to push out the new prefix list and consequently receive route refreshes from the neighbor based on the new prefix list. The keyword prefix-filter is ignored if the prefix-list ORF capability has not been received from the neighbor. Without the keyword prefix-filter, the command...

Rate Limiting with CAR

It is an inevitable consequence of being part of the Internet that every ISP at some point will experience a DoS attack. It is imperative that ISPs have tools and procedures in place to respond to these DoS attacks, preferably before an attack occurs. CAR is one such tool. CAR is a functionality that works with Cisco Express Forwarding, found in IOS Software Releases 11.1CC and onward from 12.0. It allows network operators to rate-limit certain types of traffic to specific sources or...

RCMD to the Router

RCMD requires the operator to have the UNIX rlogin rsh clients to enable access to the router. Some ISPs use RCMD for grabbing interface statistics, uploading or downloading router configurations, or taking a snapshot of the routing table. The router can be configured so that RCMD connections use the loopback interface as the source address of all packets leaving the router Configuring interfaces involves more than simply plugging in the cable and activating the interface with the IOS Software...

Redistribute static into BGP

Using the redistribute static command injects all static routes into BGP. Again, the router periodically must examine the RIB to see if there are any additions to or deletions from the static route configuration. This takes extra CPU cycles. And if the next hop to where the static route points disappears, the static route is withdrawn, with a resulting withdrawal from BGP. A withdrawal from BGP results in a route flap, as mentioned previously. (As in the IGP case, the permanent static route is...

Remote Access

For ISP engineers, the best way of accessing a network in recent years has to be use Secure Shell. In the early days of the Internet, Telnet was quite popular, but it now has largely been abandoned in the developed Internet as being far too insecure and risky to use. Communication between host and client is unencrypted passwords to log in are unencrypted. Anyone sniffing packets or snooping on a network has immediate access to the network, a risk that is too big for most ISPs. The preferred...

Route Refresh Capability

The route refresh capability is a long-awaited technique in the operator industry that lets eBGP (and, indeed, iBGP) neighbors apply a new policy to the BGP session without tearing down the whole peering. For many years, the only technique available when an ISP wanted to apply a new policy to a peering session with a neighbor was to shut down the BGP peering and then bring it up again. Consider what happens in the Internet when this is done Shutting down a peering means that all the prefixes...

Router Command Auditing

Suppose that it has been a bad day at the office. You have had to fire an engineer on the operations team. It was a rough experience, with words exchanged. You have briefed your team, and people are starting to change things on the network, removing the ex-employee's access. Suddenly alarms trigger on all sorts of equipment in the NOC. A network-wide outage is spreading fast. Is it an attack Is it a routing protocol collapse What is happening Devices are failing throughout the network, and...

Salsa ACLs in the Cisco 12000 Engine 1 Line Card [9

The Salsa ASIC is a specialized chip that assists the line card's CPU in packet-processing (features) and route-lookup (forwarding) functions. Ingress (input) ACLs are the key packet-processing feature that the ASIC optimizes. By doing the input ACL lookup as it does route lookup, the Salsa ASIC frees up the line card's CPU, allowing for considerably faster packet processing. Because this is only a firstgeneration implementation of ACLs in ASICs, some limitations still exist Only ingress ACLs...

Scion NetScarf

This was a project by Merit Network, Inc., to get a picture of what is happening on the Internet. Apart from versions that run on most UNIX platforms and their variants, there is a version that runs on Windows NT. The most recent version of the software was released in June 1997. The project has now ceased because funding has run out, but the web site and software are still available http www.merit.edu netscarf . NeTraMet is one of the original and better tools for TCP IP flow analysis. SingNet...

Scotty Tcl Extensions for Network Management Applications

Scotty is the name of a software package that allows implementation of site-specific network management software using high-level, string-based APIs. The software is based on the Tool Command Language (Tcl), the latest source for which can be found at http www.scriptics.com , and it is now part of most Linux distributions. Tcl simplifies the development of portable network management scripts. The Scotty source distribution includes two major components. The first is the Tnm Tcl Extension, which...

ShowIDB

Each interface on the router has an associated interface descriptor block allocated to it. In the early days, each physical interface mapped to one IDB, and routers generally could support up to 300 IDBs (for example, the Cisco AGS+). However, with the increasing numbers of new connection services, and with ATM and Frame Relay providing large numbers of subinterfaces, routers have had to scale to supporting several thousand IDBs. show IDB recently has become a visible command in IOS Software...

Simple Network Plan

Figure C-1 shows a simple network diagram of a basic ISP point of presence (PoP), which will be used in these examples. It has the key elements of an ISP PoP a border router, two core routers, aggregation routers (for leased-line or permanently connected customers), two service routers (for web hosting and the ISP's own services), a dial aggregation router, and a router that connects to the network operations center. Obviously, as ISPs grow, their network will be more sophisticated than this,...

SNMP in Read Write Mode

If SNMP will be used in read-write mode, think very carefully about the configuration and why there is a requirement to do this. Configuration errors in this scenario could leave the router very vulnerable. If possible, put an ACL at the edge of your network to prevent outside parties from probing your network with SNMP. Many publicly and commercially available tools will scan any network on the Internet with SNMP. This could map out your entire network or discover a device that has had SNMP...

Some Examples

The first policy decision that an ISP will need to make is what to do with the packets when an attack is identified. When DDoS attacks are identified and classified, the ISP must make a decision. Classification will provide the DDoS flow's source IP addresses hence, a target for filtering. However, do you filter all the packets from that source address, filter some of the packets from that source address, or just rate-limit the attack from that address The source IP addresses from DDoS attacks...

Syslog Source Interface

Syslog servers also require careful protection on ISP backbones. Most ISPs prefer to see only their own systems' syslog messages, not anything from the outside world. Denial-of-service attacks on syslog devices are not unknown, either. Protecting the syslog server is again made easier if the known source of syslog messages comes from a well-defined set of address space for example, that used by the loopback interfaces on the routers. See the following configuration example logging buffered...