Long before SDN entered the vernacular of most IT professionals, managers of one of the world’s oldest, largest and most geographically distributed data networks, AT&T, were building a new software-based architecture. The goals: accelerate the release of network services and integration of new equipment, while also simplifying the design and lowering costs, all without compromising the network’s stellar reputation for reliability and security.
Initially targeted at whittling an unwieldy number of suppliers to a manageable few, the plan would enable a global-scale software-defined network architecture and a transition to network functions virtualization (NFV), moving from hardware- to software-based network services. The software-centric design would allow any Tier 1 service provider’s infrastructure and network services to be used, provisioned, and orchestrated like a cloud service, and it foreshadowed the SDN and network virtualization projects many enterprises have recently embarked upon.
Although AT&T’s network, scale, requirements, and legacy infrastructure are unlike those in the typical enterprise, the complexity, interdependencies, and demands of a large, transformational project are quite similar to those of major network redesigns or refreshes now underway in many commercial organizations across the globe. As a senior leader in AT&T for many years, I knew, as did my colleagues, that making sweeping changes to a network of such size without disrupting existing services is asking to try to rebuild an airliner in mid-flight and change all engines while transforming. Simply stated, it is not an easy task to accomplish. Although such a feat isn’t possible for Boeing, it’s entirely feasible for network operators, at least with the proper planning, project governance, and task automation.
A Journey of 1,000 Miles Begins with a Single Step
Whether it’s Domain 2.0 or a new enterprise SDN, the biggest impediments to success for large network projects are their sweeping scale, myriad interdependencies, variety of stakeholders, and effects on legacy systems and services. As network managers and architects, it’s easy to get so transfixed by the big picture that you suffer from paralysis by analysis. Although it seems counterintuitive, it’s better to focus on the trees rather than forest. Identify a number of intermediate steps, each with clear, measurable objectives that build toward the overall goal. As they say, the only way to eat an elephant is one bite at a time, so rather than biting off one big project, break it down into smaller chunks that can be easily tracked and completed each quarter. If it’s a five-year project, make it 20 quarterly plans, what we called Q-Plans. The probability of success is inversely proportional to the size of the project, so it’s important to simplify each Q-Plan into tasks that can be reasonably completed within a quarter.
Each bite-sized Q-Plan should have clearly defined and agreed-upon starting and ending points, with objective measures of progress and success. These KPIs (key performance indicators) which can easily be coupled with Key Security Indicators (KSIs) and should be easy for project members to understand, easy to track for progress, and critical to the success of each quarterly sub-project. “Key” is the operative word: Adopt too many and people begin to lose focus. KPIs and KSIs should also be quantifiable and measurable so that regular reviews can quickly show whether the project is making progress. Only after meeting each of the endpoint KPIs can the project move on to the next Q-Plan.
But the Q-Plan is still too broad in scope to identify specific tasks and deliverables. For that you need a timeline, what we called a T-Plan, that moves from the 10,000 foot Q-Plan view to the 10-foot action-item level. The T-Plan should identify individual tasks and who is responsible for each; accountability is imperative. Break the project into distinct categories, like device requirements and management, network monitoring and management, service automation and orchestration, or customer support — and make someone responsible for achieving the KPIs in each area. Whereas the strategic plan and incremental Q-Plans focus on the network services you want to deliver, the what, while T-Plans will identify the necessary technologies, software and processes, the how.
Throughout the project, it is important to constantly consider the operational implications of everything you do and to carefully plan the migration from old to new technologies, because unless you’re building a greenfield environment, there will be overlap. Similarly, network security must be considered and built-in at every step of the project, otherwise small oversights today will become problems of Biblical proportion a few years from now. Just look at the recent Heartbleed vulnerability for a textbook example of how a small oversight can have incalculable ramifications.
Don’t Reinvent the Wheel: Reuse, Automate, and Collaborate
Recycling isn’t just good for the environment; it’s the best way to make big projects smaller. Don’t reinvent the wheel; in fact don’t invent it at all if there’s an existing third-party product or service. By aggressively moving to cloud services, reusing existing code and automating internal processes, you can reduce the time to bring a new network service online by 50 to 90 percent.
Collaboration, both within your project team and with other IT practitioners facing similar challenges, is another great way to both eliminate redundant work and learn from others’ experiences.
Break Big, Intractable Problems into Manageable Pieces
Successfully building a new software-defined network need not be a career threatening exercise; in fact it can be an enriching career highlight if you follow the right steps:
- Break big projects into small pieces
- Assign accountability and measure using KPIs
- Embrace the concept of Zero: eliminate the manual steps in an end-to-end process through systemic automation.
- Collaborate with peers
- Reuse tested coded and processes, wherever they might come from
- Adopt a culture of success. Don’t let naysayers defeat the project.
- Successes breeds’ success: build on small victories. Once you get a few quarterly achievements under your belt, it builds confidence within the team and with upper management.