ISP Essentials
Introduction
The Internet economy has played a significant part in the world economy since the mid 1990s. For many years prior, the Internet was the domain of U.S. academic research and defense internetworking, and a few entrepreneurs around the world who believed that a TCP IP-based wide-area network (WAN) would be a viable alternative to the private wire networks that businesses were using to communicate with each other. The many ISP engineers who learned their skills in that period look on those early...
Redistribute static into BGP
Using the redistribute static command injects all static routes into BGP. Again, the router periodically must examine the RIB to see if there are any additions to or deletions from the static route configuration. This takes extra CPU cycles. And if the next hop to where the static route points disappears, the static route is withdrawn, with a resulting withdrawal from BGP. A withdrawal from BGP results in a route flap, as mentioned previously. (As in the IGP case, the permanent static route is...
BGP Dynamic Reconfiguration
Two methods are available now to dynamically reset a BGP peering session without tearing down the entire peering. Normally, when an ISP requires changing the policy in a BGP peering, the peering itself has to be torn down so that the new policy can be implemented. For peerings exchanging a large number of routes in the Internet, this can be extremely disruptive, putting load on the CPU of both routers involved and resulting in a routing flap through the backbone as the ISP's network...
BGP Route Flap Damping
Route flap damping (introduced in Cisco IOS Software at Release 11.0) is a mechanism for minimizing the instability caused by route flapping. A route flap occurs when a BGP network prefix is withdrawn and reannounced specifically, this happens when a BGP speaker hears a WITHDRAW followed by an UPDATE for a prefix. (A peering with an eBGP neighbor being reset does not count as a flap.) Whenever a network goes down, the rest of the Internet is told about it. Hence, BGP propagates this state...
Conditional Advertisement Example
Consider the example depicted in Figure 3-8. This shows a dual-homed enterprise network (AS 300) that has received address space from its two upstream ISPs. It announces the 215.10.0.0 22 prefix to ISP 1 (AS 100) and the 202.9.64 23 prefix to ISP 2 (AS 200). These networks are part of the respective upstream ISPs address blocks, so all that the Internet sees are the two aggregates as originated by ISP 1 and ISP 2. This is the steady state situation. Figure 3-8. BGP Conditional Advertisement...
BGP Flap Damping
A consequence of the churning in the Internet caused by indiscriminate clearing of BGP sessions, by unstable infrastructure, and by antisocial configuration practices is the request for BGP flap damping to be available in IOS Software. Prefixes that appear and disappear from the Internet Route Table cause a CPU hit on the router Withdrawing a prefix means that it has to be withdrawn from the BGP and forwarding tables the prefix also has to be withdrawn from neighbors and so on throughout all...
Access Lists on the VTY Ports
It is important to secure the VTY ports used for Telnet access with a standard ACL. By default, there are no access controls on any of the VTY ports. If this is left this way and a password is applied l to the VTY port, the router will be wide open to anyone who attempts a brute-force crack against the password. The following configuration with access-list 3 is typical of a better approach aaa authentication login Cisco-Lab local username Ciscol password 7 11041811051B13 access-list 3 permit...
Some Examples
The first policy decision that an ISP will need to make is what to do with the packets when an attack is identified. When DDoS attacks are identified and classified, the ISP must make a decision. Classification will provide the DDoS flow's source IP addresses hence, a target for filtering. However, do you filter all the packets from that source address, filter some of the packets from that source address, or just rate-limit the attack from that address The source IP addresses from DDoS attacks...
Primary and Backup Paths
The first example considered (shown in Figure 5-23) is one with two paths between the networks One path is used as the primary link, and the other path is used exclusively for backup. This situation is used commonly when the primary path has a high bandwidth and the backup path is of low bandwidth or poor latency and is sufficient only when the main link has failed. Figure 5-23. Primary and Backup Paths to the Same ISP Figure 5-23. Primary and Backup Paths to the Same ISP The primary path is...
ICMP Unreachable Rate Limiting
DDTS CSCdp28161 added an ICMP Unreachable rate-limiting feature in IOS Software Release 12.0(8)S. This feature is turned on by default and is not seen in the CLI unless the default rate limit is changed. This feature rate limits the router's responses to ICMP Unreachable messages leaving the router. The objective is to keep the router from being overwhelmed with an ICMP Unreachable overload (that is, a DoS attack against the router). By default, the router rate limits ICMP Unreachables to one...
Black Hole Routing as a Packet Filter Forwarding to NullO
Another way of implementing destination-based packet filtering on a router is to create a specific list of static host routes and point them to the pseudo-interface NullO. This technique commonly is referred to as black-hole routing. Null0 is a pseudo-interface, which functions similarly to the null devices available on most operating systems. This interface is always up and can never forward or receive traffic. Although Null0 is a pseudo-interface, within CEF it is not a valid interface....
CLI String Search
After a considerable number of requests from ISPs, a UNIX qrep-like function (pattern search) has been introduced as a new feature in IOS Software from releases 11.1CC and 12.0. It allows operators to search for common expressions in configuration and other terminal output. Again, only salient points are covered here because the IOS Software documentation now gives more detailed information at 20t1 cliparse.ht . The function is invoked by using a vertical bar , like the UNIX pipe command begin...
The Zero IP Subnet
The command ip subnet-zero tells the router that the zero subnet is a legitimate subnet of the classful network being configured on the router. For example, if the Class C network 129.168.128.0 is subdivided into eight subnets, each would have a 27 mask for the zero subnet, 192.168.128.0 27, through to the seventh subnet, 192.168.128.224 27. The first and last subnets of a classful network historically were not used because of the potential confusion between these and the network broadcast...
Network Plan End of First Year
The final stage to consider is the plan at the end of the first year of operation. Again, this is a projection on what the business could be like. It is assumed that the ISP has started to do Web hosting for its customers and is investing in a large dialup network (which we shall say that the business plan calls for). The ISP also plans to deploy two of everything by the end of the first year of operation. This affords increased redundancy and resiliency in the network. (This isn't meant to be...
Example C5 NOC Router Configuration Example
Service timestamps debug datetime msec localtime show-timezone service timestamps log datetime msec localtime show-timezone aaa authentication login default tacacs+ enable aaa authentication enable default tacacs+ enable aaa accounting exec default start-stop tacacs+ aaa accounting commands 15 default start-stop tacacs+ clock timezone GMT 0 ip subnet-zero no ip source-route no ip finger ip telnet source-interface Loopback0 ip tftp source-interface Loopback0 ip ftp source-interface Loopback0 ip...
SNMP in Read Only Mode
If SNMP is used in a read-only scenario, ensure that it is set up with appropriate access controls. The following is an example access-list 98 permit 215.17.34.1 access-list 98 permit 215.17.1.1 access-list 98 deny any snmp-server community 5nmc02m RO 98 snmp-server trap-source LoopbackO snmp-server trap-authentication snmp-server enable traps config snmp-server enable traps envmon snmp-server enable traps bgp snmp-server enable traps frame-relay snmp-server contact Barry Raveendran Greene...
Route Reflectors
The first essential feature is the route reflector. Route reflectors are part of the BGP standard and are described in RFC 2796. It is generally recommended that new ISP network installations today install route reflectors from day one. Failing that, it is also very easy to migrate from using a full iBGP mesh to using route reflectors. Referring to Figure 5-1, the general design principle is to set up the core routers as the route reflectors and have the remaining routers in the PoP configured...
Example 45 Multihomed uRPF
Interface serial 1 0 1 description Link to Acme Computer's Router C ip address 192.168.3.2 255.255.255.252 ip verify unicast reverse-path no ip redirects no ip directed-broadcast no ip proxy-arp ip route-cache distributed neighbor 192.168.10.3 remote-as 65000 neighbor 192.168.10.3 description Multihomed Customer - Acme Computers neighbor 192.168.10.3 update-source Loopback0 neighbor 192.168.10.3 send-community neighbor 192.168.10.3 soft-reconfiguration inbound neighbor 192.168.10.3 route-map...
Network Plan Starting Off
The first stage involves looking at the network design at the start of the ISP's operation. Figure 5-12 gives an example network it has four routers, three switches with some hosts connected to them, and some customer leased-line connections. There is also a dialup router. Finally, the network has a link to an upstream ISP. This is a simple network with four small PoPs at initial rollout. Figure 5-12. Network Plan at Deployment Figure 5-12. Network Plan at Deployment Also on the figure are the...
Multihomed Stub Network
If the network in the preceding example is multihomed on to their upstream by connecting to different routers in their upstream's network, BGP will have to be used. Some ISPs use an IGP for this function (we have seen RIP, EIGRP, and OSPF all being used), but this practice is strongly discouraged because it is a very serious potential source of misconfiguration and problems in the ISP backbone. The chances of having the customer's IGP leaking into the ISP's IGP are very great such...
One Upstream ISP and One Local Peer
To start the list of examples, consider perhaps the simplest and most common multihoming that a small ISP will implement, as shown in Figure 5-28. The ISP connects to its upstream ISP, and it needs to connect to its local (competing) ISP so that domestic traffic does not use the expensive upstream connections. Figure 5-28. One Upstream ISP and One Local Peer Figure 5-28. One Upstream ISP and One Local Peer The common way in which this is configured today is for the upstream ISP to send the full...
Multiple Upstream ISPs and IXP
The third example adds a little more complication to the network and is probably one of the more extreme cases found on the Internet today, as shown in Figure 5-30. This is an example of a well-connected ISP, present at the local exchange point, with a few private peers, some regional ISP connections providing backup transit, and a couple of upstream Tier 1 ISPs providing Internet transit. The configurations for this aren't too hard to generate, either. Following the previous examples, the...
SNMP and Commercial Network Management Software
One thing to be aware of is that some commercial network management software likes to take over the network by doing autodiscovery of devices on the backbone. Many ISP engineers don't approve of this style of network management and tend to build their own tools that are suitable for monitoring the backbone. For those ISPs that rely on commercial packages, such as HP OpenView, it is worth remembering and understanding the impact that the autodiscovery function has. Autodiscovery fits very nicely...
Configuring OSPF
OSPF enforces a fairly rigid design for an ISP backbone. Area 0 is the backbone area and must exist if there are to be more than two OSPF areas in a network. Area 0 provides transit between the other areas, and every other area must be connected to it. OSPF offers a multitude of area types, including backbone, regular, stub, totally stubby, and not so stubby areas. Most ISPs tend to use only backbone and regular areas very few make use of OSPF inter-area summarization capabilities. The reason...
Ciscocertified Mail
Mail is the second most important service on the Internet after Web browsing. Journalists always talk about surfing the Web most new PC purchases are the result of people wanting to dial up and look at Internet Web sites to find information. Keeping in touch is the second major reason for buying a PC, and e-mail is the popular way of doing so. Also, many companies now use e-mail as the primary method of distributing information to customers and distributing information internally. So, if e-mail...
ISP Border Packet Filters
The decision of whether to install packet filters on the network border usually depends on the size of the ISP operation and whether the design and operations team feels that such filters can serve any useful purpose to protect the backbone. There is no hard-and-fast rule, but we have found that smaller ISPs tend to implement quite severe filters on their network edges, while the largest ISPs probably implement only one or two key filters to prevent DOS attacks on their networks. Much of any...
Configuring EIGRP
EIGRP also is used quite extensively in ISP backbones, finding favor with ISPs that have been required to support multiple protocols in the past. It's also accepted that EIGRP is probably the easiest routing protocol to get started with. It found favor (in the form of IGRP) with ISPs that didn't want to use IS-IS in the earlier days of the Internet when implementations of OSPF were still not mature enough for their needs. EIGRP has no area concept the network runs as one large IGP. EIGRP does...
Address Space for Customers
Customers are assigned address space by the ISP according to the policies set out by the RIR of which the ISP is a member. As an approximate guide, ISPs are expected to follow the same procedures with their customers as they have followed with the RIR in obtaining address space. So, when the ISP makes first technical contact with the customer, an assessment of address space requirements is made according to need. Need should not be confused with want. Staff members from many organizations who...
Copying New Images to Flash Memory
Copying a new image into Flash memory in itself isn't a complicated process, but there are a few good practice points to be aware of. The most important point is to re-emphasize that leaving a backout image somewhere on the router is good practice and plain common sense. So many network outages have been prolonged because a new router image failed and the ISP didn't leave a backout image on the device. New images should be loaded into Flash during maintenance periods, not when the router is...
Figure 48 Ingress Filtering
Egress filtering applies a filter for all traffic leaving an ISP's networks (see Figure 4-9). It is applied to information leaving the network to the Internet or customer networks. Be mindful that these terms are relative to the specific network's point of view. For example, ISP B's egress traffic is ISP A's ingress traffic. Ingress egress filters help protect an ISP's resources and its customers' networks, allows it to enforce policy, and minimizes the risk of being the network chosen by...
Using ACLs for Ingress Packet Filtering Preventing Reception of Invalid IP Addresses
Ingress packet filtering validates the packets from the outside world (ISPs and customers) into and across your network. For an ISP, the outside world is any place outside the ISP's control. Obviously, packets from other ISPs are from the outside world and are not to be implicitly trusted. This also means that packets from an ISP's customers are from the outside world. Just because a network is a customer of an ISP does not make it a trusted network. For ISPs that provide service to end...
Committed Access Rate CAR
Configuring Committed Access Rate c f qcprt1 qcfcar.ht RFC 1812. Requirements for IP Version 4 Routers. F. Baker (ed). June 1995. (Status Proposed standard.) Also see the update, RFC 2644. RFC 2196 FYI8. Site Security Handbook. B. Fraser. September 1997. (Obsoletes RFC 1244) (Status Informational.) One of the most useful starting places for Internet security. RFC 2827 BCP 38. Network Ingress Filtering Defeating Denial-of-Service Attacks Which Employ IP Source Address Spoofing. P. Ferguson and...
Distribution
The distribution layer is one step removed from the core and gets its name from its function of acting as a distribution layer between the core routers and the access part of the network. Indeed, many small- to medium-size ISPs don't have any distribution layer they simply connect the access part of the network to the core. It all depends on the size of the PoP. The distribution layer can be made up of two or more routers quite often there could be considerably more. ISPs conscious of providing...
The Old Classful Route Lookup Rules
The old behavior of a Cisco router is to assume that it knows about all the routes subnetworks of a directly attached network. It will do a classful route lookup to determine a match and then check for subnets. For example, when the router receives a packet, the network portion of the destination address is compared to the routing table for a match. If there is a match, the subnets listed for that classful network are examined. If a match is found, the packet is forwarded. If no match is found,...
The BGP Best Path Algorithm for IOS Software
Assign the first valid path as the current best path. Now compare the best path with the next path in list, until the end of the list of valid paths is reached. 1. Prefer the path with the largest weight. Note that weight is a Cisco specific parameter, local to the router on which it is configured. 2. Prefer the path with the largest LOCAL_PREF. 3. Prefer the path that was locally originated through a network or aggregate BGP subcommand or through redistribution from an IGP. 4. Prefer locally...
Details Behind uRPF Multihomed Customers and Asymmetrical Routing
Understanding what is happening with routing on the Internet is essential to the configuration of uRPF strict mode on multihomed leased-line customers. For starters, realize that asymmetrical routing is very common for a multihomed leased-line customer (see Figure 4-24). When traffic travels over the Internet asymmetrically, it usually means that packets will take one path to get to the destination and another path to return to the source from the destination. TCP IP, of course, works perfectly...
Business Model and IP Address Space
It generally is accepted that to think ahead one and two years when designing an ISP network is reasonable. Beyond that, given the rapid growth of the Internet, it is very difficult to predict what new technologies will be available or which direction the business might be headed. In the early 1990s, available address space was more limited than it is today. CIDR hadn't yet been deployed on the Internet backbone, and Class B space was rapidly running out. There was real concern about the rate...
Route Refresh
A new feature available from IOS Software Release 12.0(5)S is route refresh (documented in RFC 2918). The concept is similar to soft reconfiguration, but this is a capability shared between two BGP speakers (as opposed to soft reconfiguration, which is configured on the local router only) and it is negotiated automatically at the time the BGP session is brought up. To find out whether route refresh is supported, check the BGP neighbor using the following command alpha> sh ip bgp neighbors...
Figure 31 Dual Gateway LAN
Interface ethernet 0 0 description Server LAN ip address 169.223.10.1 255.255.255.0 standby 10 ip 169.223.10.254 interface ethernet 0 0 description Server LAN ip address 169.223.10.1 255.255.255.0 standby 10 ip 169.223.10.254 interface ethernet 0 0 description Service LAN ip address 169.223.10.2 255.255.255.0 standby 10 priority 150 standby 10 preempt standby 10 ip 169.223.10.254 The two routers have their LAN IP addresses conventionally defined in the preceding configuration. However, another...
Networks That Should Not Be Advertised on the Internet
As mentioned earlier, some networks are reserved for special functions on the Internet. These networks should not appear in the Internet Route Table examples are 0.0.0.0 0 and 0.0.0.0 8 Default and network 0 (unique and now historical properties) 127.0.0.0 8 Host loopback 192.0.2.0 24 TEST-NET generally used for examples in vendor documentation 10.0.0.0 8, 172.16.0.0 12, and 192.168.0.0 16 RFC 1918 private addresses 169.254.0.0 16 End-node autoconfiguration network in the absence of DHCP Any...
Aggregation Router Filters
The minimum inbound filter that any ISP should be applying to the customer-facing interfaces on its aggregation routers is the unicast RPF check. This ensures that all packets coming from the customer are checked to make sure that their origin address comes out of the address block assigned to the customer. This check is much more efficient than applying any inbound filters on the aggregation router, and it is a recommended best practice throughout this book. If the customer has address space...
Command Group Organization
The commands listed in the first preceding group have no ambiguity, so they all can appear under the router bgp < as> definition. The commands listed in the second group have no ambiguity either, so they can follow the first group of commands under the main BGP definition (with the exception of VPNs). The commands listed in the third group can have potential ambiguity and thus are listed under a new address family submode A neighbor can have different route map or prefix list statements,...
IGP Adjacency Change Logging
Neighbor state logging should be enabled in each IGP. This means that it becomes easier to find out about neighbor states, reasons for state changes, and so on. For each IGP, the IGP subcommand log-adjacency-changes enables logging. (Some older versions of IOS Software require the IGP to be specified as well for example, ospf log-adjacency-changes.) By the time this book is published, the command will likely be enabled by default in IOS Software. If logging is enabled, log messages are sent to...
NTP in a PoP Example
Devices in an ISP PoP do not need to be part of the backbone NTP mesh. Instead, the devices in the PoP (routers, NAS, switches, and workstations) use the two core PoP gateway routers as the NTP servers for the PoP. All devices will use both routers as NTP sources, simplifying the NTP configuration and decreasing the NTP convergence time in the PoP. As can be seen in Figure 1-4, devices in a PoP all need time synchronization. Accounting on the RADIUS server needs to be synchronized with the NAS...
Simple Network Plan
Figure C-1 shows a simple network diagram of a basic ISP point of presence (PoP), which will be used in these examples. It has the key elements of an ISP PoP a border router, two core routers, aggregation routers (for leased-line or permanently connected customers), two service routers (for web hosting and the ISP's own services), a dial aggregation router, and a router that connects to the network operations center. Obviously, as ISPs grow, their network will be more sophisticated than this,...
Figure 430 How Smurf Uses Amplifiers
Each host on that IP network will take the ICMP echo request and reply to it with an echo reply. This multiplies the inbound traffic by the number of hosts responding. On a multi-access broadcast network, potentially hundreds of machines could be replying to each packet, resulting in what is called an attack from a smurf amplifier network. The systems most commonly hit by these types of attacks are Internet Relay Chat (IRC) servers, specific Web sites, and their providers. Two parties are hurt...
Multihomed Leased Line Customers Two ISPs
URPF also works with a multihomed customer that has a connection to two different ISPs. Figure 4-29 shows how the downstream customer (enterprise or ISP) connects to two upstream ISPs. These two ISPs, Alpha and Beta, interconnect with each other at various places in the world (combining private peering, IXP peering, and transit). Therefore, each ISP will have two BGP entries for the downstream customer's prefix. Yet each ISP would select the shortest-path entry from the BGP table as the best...
Working Example of uRPF Multihomed Customers and Asymmetrical Routing
In this example, depicted in Figure 4-28, the enterprise customer of the ISP is multihomed into two different routers. BGP is used with a private ASN assigned by the ISP. The enterprise's IP address block would be allocated from the ISP or from an IP registry. As the route is advertised into the ISP's routers, an internal BGP weight is applied. This ensures that if there is a tie between two identical prefixes, the one directly from Router C will be preferred on the local router and entered...
ISP Community Usage
RFC 1998 was written several years ago, and since then, ISPs have refined and enhanced what they use communities for. Many examples exist on the Internet, and a few of them that were publicly visible at the time of this writing are documented here. The first example is from AS 2764, an Australian-based ISP. The community policies are documented in the AS object stored in the Internet Routing Registry Announce to non customers with no-export changed mrp connect.com.au 19990506 The remarks...
Net Flow Data Export
The greatest benefits of NetFlow are found when its data is exported to collection systems and then are analyzed and processed. Cisco has adopted a broad approach to facilitate this activity. These include donations for freeware collection analysis software, Cisco's own commercial software, tools for others to create their own software, and partnerships with companies that make commercial-grade billing systems based on NetFlow export. To export the data, the following configuration commands are...
No IP Unreachables
For a long time, Cisco routers had the configuration capability to turn off ICMP Unreachable response. This was done with the interface command no ip unreachables. Whether this is done is an operational decision of the ISP some do and some do not. The router requirements RFC (RFC1812) says that each device should respond with ICMP Unreachables, but when a network operator experiences an attack against a router, RFC niceties get left behind. What can be recommended or considered is that no ip...
Ip unnumbered
Traditionally ISPs have used IP addresses for the point-to-point links on leased-line circuits to customers. Indeed, several years ago, before the advent of CIDR, it was not uncommon to see a 26 or even a 24 used for simple point-to-point link addresses. With the advent of CIDR, 30 networks have been used instead ( 30 is a block of four addresses, two of which can be used for physical interfaces). However, this led to problems because IGPs of some of the larger ISPs were starting to carry...
What Other ISPs Are Doing
Here are just a few examples of what ISPs from all over the Internet are using to manage their network. Randy Bush (randy psg.corr) asked major ISPs in the United States on the NANOG mailing list what they used for traffic analysis. His summary follows. Notice especially the number of UNIX script-based tools. Readers can find more information by looking at the NANOG mailing list archives at www.nanog.org. We do SNMP polling every 15 minutes at SESQUINET on every line over which we have...
Console Server
The next logical step from using one modem or multiple modems is to install a console server. This is a device to which all the equipment consoles are connected. The most popular console server in recent years has been the Cisco 2511 router it has 1 Ethernet, 2 WANs, and 16 asynchronous serial ports. Originally intended as a dialup router, it has long since been retired from that function in most ISPs and now is used as a console server throughout many ISP backbones. The configuration is...
Show interface stats
The IOS Software command show interface stats is the second useful command to show interface status. It shows the number of packets and characters inbound and outbound on an individual router interface or all of them. The full command format is show interface int n n stats, where an optional argument is the specific interface in question. Command completion cannot be used for stats at least st needs to be typed in at the command prompt. Sample output might look like this Interface FastEthernet0...
IP Prefix List Example
Prefix lists also can be used and, indeed, should be used. The preceding example was rewritten using the ip prefix-list commands available in Cisco IOS Software versions 11.1CC and 12.0 and later software releases. This makes the configuration more readable, if not more intuitive. Again, remember to check the root nameserver networks for any changes before implementing these. It is also worth checking the BGP routing table to ensure that these networks still are announced with the following...