6Sense: Generating New Possibilities in the New Internet.
Produced by: IPv6 Summit, Inc.

The Planning, Transition, Implementation and Operation of a Multi-National IP Network
By John Lee
SVP Business Development, Internet Associates, LLC.

John Lee
Internet Associates Logo

This paper describes the author's experiences related to the architectural changes and renumbering of a multi-national fiber optic and IP based network that spanned three continents. While the renumbering was IPv4 to IPv4, the techniques and experience are directly applicable to IPv4 to IPv6 transition planning, implementation and operation. The primary difference in today's environment is due to the development and use of an IP Address Life Cycle Management solution and how it reduces the engineering planning, time and effort. It also produces graphical views, better records and relational information for continued operational management of the network.

Background
With little operational experience in large IPv6 networks, we rely on relevant experience with existing technologies that give us a window into the issues at hand such as Integrated Services Digital Networks (ISDN) and large multi-national IPv4 networks. The transition to ISDN was one of islands of (digital) ISDN that were originally interconnected by analog transmission services. BRIs are used to connect Customer Premise Equipment (CPE) to the network and PRI's are used to interconnect Wide Area Networks (WAN) and for network to network connections. For ISDN to be fully deployed it requires digital switches to replace the 1AESS, fully digital long haul facilities for PRI's and Signal System #7 for signal control in a separate control plan from voice or data.

What happened was that the replacement of analog phone switches with digital phone systems occurred rapidly over a three- to four-year period while the replacement of the long haul facilities took over 10 years and, in some cases, is still occurring today. With the development of IPv6 and the recent experience of ISDN, a great deal of care and attention went into tunneling and transition mechanisms for IPv6 to adhere to the lessons learned from ISDN. It was assumed that islands of IPv6 would spring up around the world with tunneling technology being used for an indeterminate amount of time until IPv6 transmission facilities were widely available. Instead, what has been occurring recently is the backbone or core of the WANs will be moving to IPv6 before the LANs and distribution networks, because they have fewer devices in the core and require less manual intervention and physical handling to upgrade.

IP Network Architectures
IP networks are divided into Wide Area Network (WANs) and Local Area Networks (LANs), each with core and distribution parts of the network. IP networks are architected from a physical perspective to have multiple redundant physical paths available for logical IP traffic. IP packet routing software running on the routers identifies which logical paths the packets should take to get from source to destination in the network. At any one time, the software only wants one logical path through the network to be up and passing traffic. When one physical path or router goes down the routing control software will assess the situation and re-converge on different paths for different packet streams. In large networks this process can be occurring constantly throughout the network, but network designers want these situations to stabilize to allow higher performance of the IP network. When the network is re-converging at the IP layer, packets can be misdirected or lost forcing higher level protocols to address the packet loss issue. SONET, ATM and fiber optical light frequency reroute are available below the IP layer to allow faster reroute of traffic around problems without packet loss.

Planning and Transitioning of Large IP Networks
When planning the transition of large networks the task can be divided into network issues, application servers and workstation issues. Because application transitions are beyond the scope of this paper, we will only be addressing network issues. Since IPv6 has the potential to touch every piece of network equipment, testing of IPv6 needs to occur in a lab and then afterwards in the operational test facility for the network.

Universities and other stakeholders in IPv6 developed the initial equipment and software. It was then tested in interoperability labs; both around the country and the world to better understand IPv6 issues, capabilities and limitations. The Moonv6 project at the University of New Hampshire in cooperation with DISA's JITC and DoD DREN has ably fulfilled that role in the US. Internet 2 is looking at more efficient higher performance Internet architectures that would include IPv6. My previous paper in 6Sense identified IPv6 as a when, not if, scenario. Let's begin to review specific issues regarding planning and transitioning a large IP network.

One previous IPv4 network spanned three continents, from Japan, throughout North America to Western Europe, and started out as a multiple vendor router network migrating to a single vendor router for the back bone and two vendors for distribution and aggregation of packets and flows. It was decided to handle the renumbering one continent at a time, starting with the US since the geographic and time differences were minimal. There were two NOCs in the US — Portland, OR and Houston, TX, with a small regional NOC in London. The phases of renumbering were planning, assessment, documentation and testing, implementation and verification and then operational turn-up. An initial assessment was completed to determine the scope of the project and resources necessary to complete the plan. Some issues addressed included:

  • Routers, switches and interface cards at a consistent and standard hardware and driver revision
  • Routers, switches at a consistent and standard operating system level
  • All device management ports on an Out of Band management network (OOB network)
  • Router configurations reviewed for consistent structure and commands and tested before deployment
  • Test scripts completed and tested on pre-changed networks and any differences noted after changes
  • Implementation included contingency and roll back points in case the network change did not work
  • All work was done during normal maintenance windows as to not affect customer traffic
  • One or more Network Engineers or Technicians were on site at the PoPs (Point of presence, where all the equipment and circuit terminations are located) that are being converted
  • Extra staff was at the NOCs specifically to handle conversion issues as they arose
  • The Director of Network Engineering and all staff involved in the conversion were on a conference call that lasted the entire time of the conversion until all network connectivity was restored
  • The conversion started at the Western side of the country and preceded East
  • Following the PoP interconnect architecture there was one main PoP being converted and two to four other interconnected PoPs directly interconnected to the main PoP that would also be manned during the conversion process.


IP Address Life Cycle Management
The major component missing from the planning and implementation effort was a complete IP Address Life Cycle Management solution. The project budget was over $50 million, but a complete IP Address Life Cycle solution was not available at any price. The project had multiple spreadsheets and point utilities that were utilized to renumber the routers, switches and servers. These were inadequate and could not be kept up to date and accurate over the distance and time constraints.

A complete IP address management application was required to overcome this manual process and point tool limitations, as well as enhance enabled handling of both IPv4, IPv6 and ASN networks. ASN or Autonomous System's Number is assigned to different policy authorities to manage a partial routing domain with consistent routing policies. Two underlying concepts were developed for complete IP address management, those of the Engineered IP Address and the IP Address Life Cycle (IPALC). The Engineered IP Address is a valid CIDR address that is unique within a given routing domain. The IPALC was developed to maintain the accuracy and integrity of the EIPA throughout any IP address manipulations during the entire life cycle of each individual address.

Requirements for an IPALC solution include:

  • Single enterprise wide source of Engineered IP Addresses that are up to date and accurate
  • Service Oriented Architecture with Web-based GUI and machine based XLS and XML interfaces
  • Standard database interfaces
  • Standard Microsoft Application for Government certification
  • Supports Multiple:
    • IPv4 and RFC 1918, IPv6 and ASN space both Aggregate and Non-aggregate
    • Address allocation methods with automatic address distribution
    • DNS and DHCP vendors and servers
    • DNS Zones and DHCP Scope
    • Web Browsers
    • Network Discovery Tools
    • Intrusion Prevention and Detection Systems
  • Maintains Address Blocks, Connections and Equipment at appropriate network locations
  • Provides a visual representation of the network architecture and a Graphical interactive IP Address display available for the entire IPv4 and IPv6 space
  • User accounts that allow different classes of users access to different network segments, IP address block sizes, system features and functions
  • Real Time Reports with summaries and details of network and equipment configurations, connections and address blocks

www.internetassociatesllc.com

John L. Lee at john@internetassociatesllc.com or Office – 770-495-0953, Cell – 678-488-6085

Jim Ludwig at jludwig@sesc-us.com or Office – 770-656-9626