PMCA Data Center Relocation (DCR) Pre-Planning Guide

PM Consultants & Associates, Inc. (PMCA)

1 23

The PMCA Data Center Relocation and Consolidation

Pre-Planning Guide The PMCA DCR Pre-Planning Guide is a “high-level” introduction to Data Center Relocation (DCR) and Consolidation project planning for executives and project managers. It is specifically intended to help you begin the extensive “pre-planning” activities that any organization should start prior to the detailed work of DCR planning specialists. This Guide can help reduce the time, effort, and costs of planning for a major (or smaller) data center relocation or consolidation initiative. A set of useful “DCR pre-planning” templates is also available from PMCA. For more information about our comprehensive DCR Services, contact us at: www.pmca-us.com - Phone: 1.954.962.9864 - Email: [email protected] PM Consultants & Associates, Inc. © Copyright 2007 Version 1.05 All Rights Reserved

The PMCA Data Center Relocation & Consolidation Pre-Planning Guide

1 23

Contents Get Yourself DCR-Prepared ......................................................................................................................................... 4 Data Center Relocation (DCR) Initiatives .................................................................................................................... 5 DCR Pre-Planning Is Key ............................................................................................................................................. 6 Get DCR Project Help .................................................................................................................................................. 8 PMCA’s Pre-Planning Guide........................................................................................................................................ 8 What You Should Do NOW: ........................................................................................................................................ 9 Scope Creep .................................................................................................................................................................. 9 The DCR Project from Hell * ..................................................................................................................................... 11 Analysis of Our DCR Project *................................................................................................................................... 15 Data Center Relocation 101 ........................................................................................................................................ 16 Relocation or Consolidation?...................................................................................................................................... 18 DCR & Virtualization ................................................................................................................................................. 19 Getting Started Right .................................................................................................................................................. 20 PMCA’s DCR Template Kit ....................................................................................................................................... 23 DCR Risk Awareness Planning .................................................................................................................................. 24 Ten Key “Givens” For a Data Center Relocation Project ........................................................................................... 26 DCR Pre-Planning Checklist ...................................................................................................................................... 31 DCR Project Management Checklist .......................................................................................................................... 32 The Data Center Relocation Process........................................................................................................................... 33 About Consolidations.................................................................................................................................................. 41 Consolidation Checklist .............................................................................................................................................. 42 Server Consolidation & Virtualization........................................................................................................................ 43 Special Considerations................................................................................................................................................ 43 The Ten-Step DCR Plan ............................................................................................................................................. 46 DCR Pre-Planning & Disaster Recovery / Continuation of Business (DR/COB)....................................................... 48 DR/COB for DCRs ..................................................................................................................................................... 49 Who Should Do The DCR DR/COB Plan?................................................................................................................. 50 DR/COB Methodology for DCR ................................................................................................................................ 50 Phase 1: Detailed Requirements Definition ................................................................................................................ 51 Phase 2 - Plan Development & Maintenance Program ............................................................................................... 53 Phase 3 - Proof of Concept, Research, and Implementation ....................................................................................... 54 Phase 4 - Testing/Exercising Program ........................................................................................................................ 54 Phase 5 - DR Test Plan Execution .............................................................................................................................. 54 Understanding Availability and Recoverability.......................................................................................................... 55 DCR & Data Center Capacity Planning...................................................................................................................... 56 Planning Your Data Center Relocation Project?......................................................................................................... 60 PMCA DCR “Pre-Planning Checklist - Overview ..................................................................................................... 63 Detailed DCR Pre-Planning Checklist ........................................................................................................................ 64 DCR Project Management Checklist .......................................................................................................................... 70

PMCA DCR Pre-Planning Guide, v1.05 www.pmca-us.com Phone: 1.954.962.9864 PM Consultants & Associates, Inc. © Copyright 2007 All Rights Reserved

2/71

DISCLAIMER The following information is provided solely as a high level guide. It is intended to be "one example" of requirements for a major data center relocation or large consolidation plan (DCR). It is not, by any stretch of the imagination, the only way to set up a DCR planning project, although the key points discussed in this Guide are applicable to all DCR initiatives. The PMCA DCR Guides are “living” documents, and as new versions become available, they will be sent to those who have requested earlier downloads and provided their email address.

OVERVIEW This Guide is the first in a series; it provides a high level overview of the key areas of focus for those who are planning relocation or consolidation projects. It is, of course, not a substitute for experience and judgment. The planning and execution of a major data center relocation or consolidation is one of the more challenging assignments in the I.T. field. It is not something that should be approached in any but the most careful and thorough manner. Given the scale, risks, and complexities inherent in a DCR, detailed relocation or consolidation planning is also probably not a “do-ityourself” task for any but the smallest data center projects.

PM CONSULTANTS & ASSOCIATES, INC. “Choosing the right DCR project planner and approach is perhaps the single most important thing you can do to reduce risk, stay on budget, and assure your project’s success.” That’s what a leading financial services company says. PM Consultants & Associates, Inc. (PMCA) delivers DCR projects within demanding schedule and budget constraints for Fortune 500 clients globally. PMCA also provides Senior Data Center Relocation, Migration, and Consolidation Project Planners and Managers to leading I.T. Consulting firms worldwide. PMCA Data Center Relocation and Consolidation Planning services are used by the Fortune 500, government, and leading I.T. consulting firms, because we are globally recognized DCR planning specialists.

For more information about PMCA’s data center relocation and consolidation planning services: Call us at 1.954.962.9864 Email: [email protected]


3/71

Get Yourself DCRPrepared PMCA’s DCR Assessment Services If you’re responsible for a data center relocation project, you can start by reading this Guide and customizing the available* DCR pre-planning templates. While the Guide can’t tell you how to identify, plan for, and execute the literally hundreds of inter-connected tasks necessary for a successful DCR project, it will at least let you know (at a high level) that you need to check for the key details – and what they are. There are two ways to kick-off a data center relocation or consolidation project. You can jump in with both feet and struggle to stay afloat. Or, you can perform an assessment to develop the scope, objectives, and key tasks first. We recommend the assessment. The DCR Assessment covers the initial site survey(s), project scope review, inventory process audit, DCR project management methodology customization, and other key areas to help you get your DCR project organized and started right. The DCR Assessment can help reduce the time and effort required to develop your RFP, organize the project, and identify key project resources, issues, and risks. It can also help to substantially reduce overall project costs. Depending on the project’s complexity and size, a high level assessment should take a few weeks, not several months. It should be tightly focused on the critical areas of project risk: • • • • • • •

Budget Facilities & Capacity Planning Inventories Application & System Inter-Relationships Project Management Disaster Recovery/Continuation of Business Schedule

Use this Guide to help you create a better RFP for the DCR project phases to come. Use it to help you reduce time, effort, and project costs. Use it to help you succeed at one of the great challenges in I.T.

The PMCA DCR Assessment is an effective, lower-cost requirements assessment and validation alternative to high-priced “packaged” data center consulting services from the big I.T. companies – that often rely on senior DCR planning talent from the specialist leaders in data center relocation planning and migration including, of course, PMCA. PMCA is an unbiased advisor. We don’t sell hardware, solutions, or applications – just world-class DCR services. To request more information about our cost effective, rapid DCR Assessment services, contact us: [email protected] Or, call:

1.954.962.9864 __________________

DCR In A Nutshell Highly simplified, there are three component parts to any data center relocation or consolidation: First: you define and organize the project, and gather basic information about each domain – this is the “pre-planning” stage. Second: you begin the detailed “planning” phase by collecting accurate inventories and assessing the major domains that will be impacted (applications, mainframes, servers, DASD, networks, etc.) This is the most complex and lengthy activity in a DCR. Third: you move or consolidate everything, verify that it was re-installed and operates properly, and then you close the project.

* PMCA offers a FREE DCR Pre-Planning Templates Kit to accompany the PMCA DCR Pre-Planning Guide. Please see the last pages of this document for information on how to order these useful templates.


4/71

DCR Overview Data Center Relocation (DCR) Initiatives Among the more challenging things that any I.T. executive or project manager will encounter in a long and varied career is the DCR – the relocation or consolidation of a major data center. DCR’s, by definition, are complex and expensive projects. And yet, they are amenable to significant cost savings when planned and executed well.

Planning The key to a successful DCR is planning – intuitively obvious, right? And, although they share the same basic project organization and management techniques with other I.T projects, DCR’s are not planned in quite the same ways that other large I.T. projects are. That’s because a DCR project has three notable characteristics: scale, complexity, and risk.

Scale

1 23

Pre-Planning Planning Move I.T.

The scale of a major data center relocation or significant consolidation is encompassing, touching virtually every part of the organization. Analogous to performing open-heart surgery while the patient is running a marathon, a DCR is usually more complex, time consuming, and costly than virtually anything else a corporation’s I.T. group will do.

Complexity The DCR will impact virtually every aspect of the business. It will affect almost every type of system and infrastructure within and touching the organization. It will involve, at some time or another, just about every technical staff member in the I.T. group. It may also involve and frustrate business unit managers, end users, and customers alike if it isn’t planned and executed well.

Risk The risks presented by a data center move or consolidation are also significant. Mishandled, a DCR can seriously disrupt current business operations and bring about regulatory and legal action. In worst-cases, the business can experience significant and prolonged operational failure.


5/71

DCR PrePlanning Is Key Essential to the success of any DCR initiative is planning. While this is, as we continue to emphasize, intuitively obvious, it is the most difficult (and too often hurried) aspect of this kind of I.T. project. The sheer size, complexities, and scope of a modern DCR can very quickly overwhelm normal project initiation and planning methods. That is why “pre-planning,” in the context of a data center relocation or consolidation, is an essential and cost effective first step.

DCR Pre-Planning Role I.T. professionals are very good at planning activities. But, a DCR demands even more – it requires “pre-planning.” Pre-planning is the set of inter-connected activities that a technically and operationally competent I.T. organization undertakes to prepare for a DCR. It begins with a detailed review and analysis of the initial business case(s) that describe the rationale and financial justifications for the project, along with a “sanity-check” on the validity of ROI projections and other outcomes that will be expected. This is the time when key team members are selected, when the basic structure and scope of the project are outlined, and when the first of many RFPs are prepared. DCR pre-planning also focuses on establishing first-pass assessments and high-level inventories of the systems and infrastructures that will be affected by the move or consolidation.

DCR Projects Are Stressful DCR Pre-Planning can really help reduce this stress by giving you a practical skeletal framework around which to build the larger project. In Fortune 500 companies and government agencies, a large DCR initiative will usually be sponsored and kicked-off by a C-Level I.T. executive – preferably the CIO. Pre-planning for this event ensures that s/he will have the right information, momentum, and team assembled to ensure the project’s success. The project’s sponsor and technical I.T. management will work together with the program/project management group (PMO) to assemble the staff, structure, and methodologies needed to initiate, plan, execute, manage, and close-out the project over its considerable lifespan. It is generally here, in the project startup phase, that the first big problems appear. The usual project initiation and management practices within the organization, while capable and essential to the success of a DCR initiative are rarely adequate by themselves to cope with the scale, speed of execution, cost, and complexities involved. The basic reason for this is that DCR’s are not “normal” I.T. projects. Nothing in the data center will touch as many people and systems. For most organizations, nothing else will be as big, as costly, or as potentially risky as a major data center move or consolidation.

Required technical and business resources, equipment and application inventories, key business processes, origin and target facilities, and external factors are all surveyed, assessed, and documented at a high level.

1

First Step: Define the DCR project scope and objectives, governance model, and high-level budget. Then begin your “Pre-Planning.”


6/71

Pre-Planning vs. Planning In a DCR, everyone plans; not everyone “preplans.” Pre-planning (the first of the three DCR phases) is the key to defining and integrating the requirements for a successful DCR project with the methodologies and skills that will be provided by the project management and technical teams. DCR Pre-planning is the set of activities and decisions that will be completed before you begin the detailed planning for the DCR project. It is what you do to prepare in advance of the DCR planning experts’ arrival, not what they do after you bring them aboard. DCR “planning” (the second of the three DCR phases) differs from the “pre-planning” phase in focus and detail. Pre-planning is a limitedscope activity that is done at a high level, and should account for about 10-15% of the total project budget for a major initiative. DCR Planning is an intensive, expensive, and highly detailed effort aimed at uncovering the key risks to the project, as well as the myriad of details that must be discovered, analyzed, verified, and documented before a single machine or application can be migrated or consolidated.

But, without adequate DCR pre-planning, it’s going to be more difficult to be sure that you have everything identified and included in the project’s scope or RFP. Without adequate pre-planning, you’re going to do a lot more work later, when the costs of omission are much greater. And that simply adds to the already considerable risk, cost, and project difficulty ahead. Who needs that?

Who’s Going to Do I.T.? Who should do the DCR pre-planning? Should you give this important task to outside consultants? Do it with internal staff? Or, perhaps you just skip it entirely, save some up-front costs, and expect the DCR vendor(s) ultimately selected through the RFP process to plan your relocation or consolidation effort? After all, isn’t that what you’re going to pay them for? Our recommendation, based on many years of global DCR experience is this: the preplanning stage is far too critical to give to anyone not intimately familiar with your I.T. environment(s).

Unless you are buying all new equipment, DCR “planning” can account for as much as 35-50% of your total DCR budget.

This means that your own internal I.T. staff, your project managers, and key application owners will all need to be involved in this very important activity.

Where pre-planning is cursory or non-existent, the detailed planning phase must make up for it. That means a lot more time and higher costs. It also implies that project risks will probably increase because early risk identification and mitigation planning is limited or absent.

Using an experienced DCR planner, a subject matter expert (SME), at this stage will, however, help you cut through the noise and chaos that comes with any major project startup.

It may also mean that the project scope isn’t adequately defined for the levels of detailed research and effort necessary. Scope (and creep-control) is a critical boundary condition for a DCR, and that depends on adequate planning.

2

The DCR SME can guide you more directly through project definition, business case development, risk assessment, the RFP process, and initial project formation faster than you can probably do it yourself. That WILL help you reduce costs and ensure the desired project outcome.

Second Step: Complete your DCR “pre-planning.” Validate the DCR project objective, leadership team, governance structure, methodologies, stakeholders, domains, and budget. Then, begin your “Detailed DCR Planning” phase with the help of DCR SMEs. PMCA DCR Pre-Planning Guide, v1.05 www.pmca-us.com Phone: 1.954.962.9864 PM Consultants & Associates, Inc. © Copyright 2007 All Rights Reserved

7/71

Get DCR Project Help You can (and should) begin the DCR preplanning yourself with the help of this Guide and the available template kit. However, your project will benefit from using a DCR planning specialist at this stage to help you shape and guide the process. This approach will also save your organization substantial time and money and reduce project risk, even at this early stage. We will keep emphasizing these points because they are critically important to C-Level management. What is also important is that with “preplanning” your organization will be able to better commit to understanding and developing the scope, cost, and high level details of what you are about to do.

Use this Guide to help you define your own DCR Pre-Planning project. • • •

The real effort will begin as the DCR planning specialists work with you to research, analyze, and document your environment in exhaustive detail. It’s not enough to collect an “inventory” and move boxes. This is not “operations.” Any major data center relocation or consolidation requires far deeper understanding of the interactions among systems, applications, business processes, and people than is required to run them. DCR-experienced I.T. professionals know that each project is unique, much more so than application development or other infrastructure projects. This merely adds to the challenges. Planning and executing a DCR project well can give your career a big push in the right direction. It can also help improve operational performance and be a path towards upgrading systems and applications that will benefit the company for years. Get it wrong, and it could be something that impacts your organization for some time.

The templates can give you a structure that will save effort The structure will help you identify and plan to mitigate key risks Together, they can help you manage the project, reduce the costs, and improve the quality of a major DCR initiative.

But, don’t be reluctant to use the services of a DCR specialist; it really will save you time and money now and improve the final project outcome. And that, as we all know, is a good thing.

PMCA’s PrePlanning Guide This Guide provides essential insight into the background activities, milestones, and risks associated with the DCR pre-planning phase. It offers an overview of the problem (moving or consolidating a large data center); guidance on defining the key tasks that make up that phase, and a set of useful templates to help get you started properly. But, please - don’t confuse the pre-planning phase of a DCR project with what comes later.

In the rush to get started in a complex DCR, the most critical mistake made (and most often) is simply under-estimating just how hard a big DCR will be. That’s not, however, a mistake that you will make. OK, the meeting has finally adjourned. YOU have been given the opportunity of a lifetime, a major data center relocation project. As you walk slowly back to your desk, you think”…now what? Where do I start?”


8/71

What You Should Do NOW: 1. 2. 3. 4. 5. 6. 7.

Identify the key stakeholders Identify the Project’s Executive Sponsor Identify the right Project Manager(s) Identify key technical leads Get a contact list of the key I.T. vendors Do a Communication Plan (first draft) Get a cup of coffee/ tea, sit down, relax…

You probably won’t get much time to do that in the weeks and months ahead. Other steps that you should take include finding out who, where and how I.T. equipment inventories are maintained. Ask what level of confidence the I.T. inventory maintainers have in the accuracy and completeness of their databases (the two are not the same). If they say that their confidence is “high,” you may be in trouble. But, more about inventories later… And one more thing: figure out where you can stash a cot and a few energy bars – for when you’ll need them.

Depending on the budget, scope, and domains involved in the DCR, you can expect business and I.T managers, end users, and even suppliers to push hard for you to (pick one or more…): • • • • • • • • • • • • • • • •

Scope Creep

• •

The DCR project is attractive to many people because it has what everyone wants: a large budget, lots of activity, cross-organization scope, and enough room to hide more than a few extras. One of the challenges that any DCR project leader will face is the pressure from just about every quarter to seize the opportunity to “fix I.T.” Not that your I.T. department is “broken” – they almost never are. But, rather, users and business managers, developers, and technical staff will all see this as a unique opportunity to push through their own departmental needs and “wish-list” items – the big ones that didn’t make it through the last budget review cycle.

3

• • •

Get more servers Get more MIPS Get more storage Upgrade & refresh “something…” Outsource and/or in-source Get rid of the “junk” and buy new Clean up those stale applications Buy new applications Retire old applications Improve or expand the network Get new network gear Offload the mainframe, servers, LAN and migrate to whatever/whoever is “hot” at the time Consolidate, migrate, re-host key solutions back on to the mainframe Cut costs by consolidating servers Cut cost by replacing old servers with the latest stuff Cut costs by reducing the number of applications being licensed Cut costs by streamlining staff Cut costs by outsourcing (fill in the blanks…) Improve customer service by… Cut high customer care expenses through… Cut costs by omitting “unnecessary” steps and resources in the DCR process…

You get the idea.

DCR Checklist At the end of this Guide, we have included a high-level DCR Pre-Planning Checklist to help you jumpstart your project. A more detailed “pre-planning” task list is provided in the DCR Template Kit in both Microsoft Excel 2003 and MS Project 2003 formats.

Third Step: Move everything (systems, applications, networks, people, etc…), verify a successful migration, and close the project


9/71

Managing Project Scope Creep Project scopes creep. They always have; they always will. Yet, the first unspoken rule about data center relocations and consolidations is that nothing important is supposed to change until the project is finished. Naturally, it’s also a rule that is widely ignored, simply because the business of doing business never stops. Your DCR “pre-planning must take this fact and the inevitable consequences of change in stride. Because a DCR project operates in a fastmoving business and technology environment, all of the systems, applications, networks, people, and architecture domains that must be analyzed are in constant motion. Your DCR planning methodology must adapt to this environment. Also, since this is probably going to be the best near-term opportunity that many groups will have to slide their new requests for hardware, software, and people into the budget this year, you are going to have to cope with the pressure to expand the project’s scope.

DCR Project Manager Responsibilities The role of the project manager in any data center relocation or consolidation is critical. It is here, more than anywhere else, that the project will success – or fail. Because the job is so important, your project managers will be among the most experienced and capable in the organization. Your project management team (and for a larger project, you will need a team) should ideally be supported by an enterprise-wide Program Management Office (PMO). The PMO should have the authority and budget to commit and manage the resources (budget, people, processes) needed to ensure the project’s success. Among the many responsibilities that the DCR project manager will have are to: • •

• Politically smart groups will quickly have their business cases and enthusiastic supporters lined up and ready for action. What this can all lead to is rapid “scope creep.” What began as a large data center relocation or consolidation project has quickly become the Mother of All Upgrades. It can only be controlled with careful planning and good project management. With good pre-planning, you can identify where the pressures for additional requirements are coming from. You can also begin to develop your strategy for staying focused on the core objective. Of course, the opposite reaction sometimes occurs: Management decrees that “NO new purchases will be made.”

•

• • •

• •

Lead meetings to manage and coordinate data center relocation, consolidation, and migration plans Lead meetings to manage and coordinate data center relocation, consolidation, and migration requirements, specifications, and architecture Lead meetings to manage and coordinate data center relocation, consolidation, and migration execution Create and maintain Microsoft Project, Excel, and Visio documentation, track and support overall project direction and progress on a daily and weekly basis Interface with construction manager(s), crew leads, and data center build teams Manage outside data center vendors and services suppliers Direct and coordinate cross-functional organizational activities for the project to ensure that key goals or objectives are accomplished Be effective working among highly technical IT professionals Manage and complete the project within an intense operational production data center environment, while using sound technical, political, and procedural judgment throughout the process. Ensure financial accountability and electronic logging for data center assets, particularly in a large enterprise I.T. environment.

Neither approach is necessarily the right one. That is why DCR pre-planning is so important. Here, you can define the scope and major objectives for the project. And work the kinks out of the DCR Budget.

•

Neglect this important pre-planning phase, and well…

See the PMI PMBOK for more information on PM roles and responsibilities.


10/71

The DCR Project from Hell * “It didn’t start out that way… When we first decided to move our primary data center to the new facility, we expected things to go smoothly. Just like before. Five years ago we had relocated the entire data center complex over a span of several weeks from one floor to another – part of a major campus redesign and build-out. That project went well, maybe too well in retrospect. Oh, we had a few problems here and there. Some DASD couldn’t be moved intact because fully-loaded rack weights exceeded floor loads in some hallways. That required us to “swing” some very expensive storage farms. The freight elevator was too small for the two big laser printers, so we had to have the vendor dismantle, move, and re-assemble them – an obvious oversight that cost much more than we’d initially budgeted for that task. They originally came into the building when the walls were opened up for the new wing, but that option was no longer available to us.

EXAMPLE CASE STUDY * This is an example case study drawn from information available about a wide range of data center relocation and consolidation projects across many industries through the past two decades. It does not reflect outcomes of any projects that PMCA has been involved in. The “DCR Project from Hell” case study is intended to illustrate common problems encountered by data center teams; and some of the key internal and external factors that can impact a DCR project’s success.

A few poorly-secured server racks fell apart intransit, several older systems failed to boot up on re-install and had to be replaced overnight. But, the virtualization products functioned as advertised, and that alone saved us many hours of work. Some of our LANs and remote nodes struggled to come up because of the usual “network” glitches. We learned again about the problems inherent in allowing application developers to “hard-code” IP addresses into their products. But all-in-all, our methodology was adequate, data recovery processes worked well, and our team elegantly handled issues as they appeared.


11/71

EXAMPLE CASE STUDY *

And so, because everything got successfully migrated and our users came up each Monday morning with a minimum of difficulty, we finally declared success and moved on. After it was all over, we conducted the usual project post-mortem, made and filed our notes, and prepared a detailed report to Management.

Making Assumptions About DCRs Our primary data center was a reasonably large facility: • •

Another successful I.T. project, some muchneeded vacation days, and everyone got a well-earned bonus. Hooray for us.

• • •

Just Another Big Project, Right?

• •

So, when the decision was made at the C-level to move to a new data center some distance away, our I.T. group felt that we could handle everything in-house. Of course, we’d grown quite a bit in those last few years, with several acquisitions, new lines of business, and a lot more Wall Street visibility. The main data center was now more than four times the size of our last move, and scattered across several buildings on and off campus. But, we were sure that our solid operational processes, tools, and talented people were more than sufficient for the job.

• • • • •

350+ enterprise & hundreds of departmental applications Twenty-six IBM e-server and p-server host complexes on three floors (two buildings) Twenty midrange AS/400’s (M&A) Seventy-five Sun Enterprise servers Several hundred terabytes of DASD, SANs, and optical drives Automated tape complex (2) Enterprise and specialized printers, mail sorters, and remittance processing equipment About 4,500 racked Wintel, UNIX and Linux servers Some 450 “box” servers, not all in the data center Technical staff desktops Networks… And people

…the usual stuff you’ll find in a large regional financial services company’s operational hub.

And anyway, we’d just done “the same thing” successfully a few years before without problems. We ‘sold” the CIO and business leads on our internal data center relocation planning and implementation capabilities, our tested DR/COB process, and got the OK to proceed with our DCR planning cycle. With the project sponsor’s concurrence, we fired up the project management team, wrangled a few more technical specialists out of the I.T. budget, added consultants, and rolled up our collective sleeves. It was an approach that we eventually came to regret.

What we all thought would be a straightforward (but, certainly not simple!) data center relocation project slowly turned into a prolonged death-march. Along the way, we lost people, machines, and data. And, we damaged our credibility. That hurt the most, I think.


12/71


The data losses were very painful because, once the extent of our problems became apparent to all; we were also hit with Federal and state inquiries and penalties, as well as the bad publicity that really hurt the business side of the company. Our DR/COB plan worked – sort of – except for the unexpected data losses. But relying on the plan that worked last time, even with all the upgrades, updates, and new off-site simulations we had acquired turned out to be less than the golden parachute we’d all hoped for.

The Capability Trap What finally tripped us up was the fact that we got over-confident. We WERE good at this! We had done several data center consolidations and moves over the last decade. Smaller, simpler ones. We brought in the major vendors and got their equipment planners involved. Our project managers all had the right credentials, had been with us for years, and knew the environment. But, somehow we failed to acknowledge the first rule of data center relocations: if you don’t plan for it, you’ll get what you do plan for. By the time we saw the train-wreck coming, it was too late to stop. Sure, this time, we didn’t have any server racks fall over (…you DO learn from some mistakes!).

As good as we were, and we were, we kept missing (and dismissing) critical information and tasks everywhere. Of course, we didn’t know that at the time.

Project Fallout We “knew” our systems and applications, and we had “documented” the key application and data inter-relationships (we thought); and we rigorously planned for every recognized alternative and risk. Although, when our automated inventory solution could not identify systems we “knew” were in the farms, we should have heeded the red flags. We did take action on many, but ignored too many others. Mostly, because we all had tough “day jobs along with the DCR stuff. Our project management processes correctly identified the basic resource and schedule requirements. We spent a third of our effort on pre-move planning, a third on the actual moves, and the rest on managing, reporting, and troubleshooting the project. But eventually, way over budget, months late, and thoroughly demoralized, we finally wrapped the project up, “declared success” and tried to move on. We’re still trying. Even now, there are systems and applications that don’t run quite like they used to. And there are more than a few business users who have been forced to move to higher-cost, outsourced solutions because they couldn’t wait for us to get the old systems fixed after the move.

This time, we got the big printers in the budget and moved properly. Our DASD farms were swung on schedule, and even the new networks (data, voice, video) got done close to schedule. The vendors all cooperated nicely. So did the weather. However, it turned out that we actually NEEDED the independent analysis, insights, and expertise of DCR specialists who do this stuff professionally.


13/71


Key Stress Points Inventory Nightmares One of the bigger problems we faced was creating accurate and complete hardware, business process, and application inventories. Sure, we kept running the automated tools. And they kept giving us different answers. We finally sent teams of our staff and contractors into the racks to physically count and audit the automated inventory tables. Server farms were always a major problem. We’ve been growing fast enough to need a new facility – that’s why the relocation. But, while we were struggling to integrate systems from various M&A’s and fast growth in our core operations over the past few years, we somehow neglected to look at our inability to accurately and completely inventory what we had. We thought we had this covered. We have and use asset management software and processes. But, when it came time to move, we had boxes and racks that we couldn’t find in the inventories. We had systems on the floor that weren’t listed accurately or at all. For many of those, we didn’t necessarily even know what was running. We never really found a perfect solution. Our tools could be tripped up by LAN configurations, or other issues. And, there were simply too many systems to physically count.

There are perhaps a few key points where the stresses in a DCR or consolidation project are most likely to occur. First: The right people aren’t involved. Determination of just who the “real” stakeholders are isn’t always obvious. Getting them to actively participate at the level of detail needed is even harder. But, it is always critical. Second: Not understanding application and system inter-dependencies at a deep enough level to limit unforeseen or “unintended” consequences. Or, not having a risk mitigation plan that can address them as they emerge. Third: Inventories don’t add up. If you can’t rely on your hardware and application inventories to accurately and completely reveal what’s in-play, you have a much bigger problem than you may yet realize. Fourth: Not adequately recognizing the critical importance of optimizing your project management processes and staff. An operationally effective and respected project management team is the single most important factor in a DCR project’s success. Fifth: Minimizing the roles and responsibilities of your key vendors, services providers, and technical DCR SMEs during the all-important “pre-planning” phase. Sixth: The project budget is inadequate, gets significantly cut, or is managed in a way that impacts the flow of activities necessary to progress and complete the effort.

So, we moved everything and segregated what we were sure of from what we were not. There are still several racks with mystery machines. We get to them as we can.


14/71


Analysis of Our DCR Project * Where we got off track was in the way we did the pre-relocation planning. Or rather, it was in the way that we “didn’t” actually do it. It finally took an independent DCR consulting firm to determine the root cause.

Application Relationships

In our case, there weren’t any “major” catastrophes. The business didn’t cease to function. Bills got paid, revenue continued to flow, and customers were satisfied.

Another critical point of failure was our documentation of application dependencies. It didn’t really exist. Not in a useful form.

The real problems were almost all internal. And they were eventually traced to inadequately prepared activity planning and task execution – we should have “pre-planned” more thoroughly. We also took a major hit because the DR/COB plan that we thought was adequate turned out to have a couple of glaring holes. But, that’s another project from hell.

Inventory Problems One of the more important tasks in a DCR project is the process of gathering and validating the inventories. There are several of them. Like most organizations of any size, we have lots of state-of-the-art tools, including automated inventory management solutions and databases for hardware, applications, key business processes, and SLAs. We work hard to keep them current, but because of the substantial costs involved, we don’t really audit these with the attention to detail that a CPA does with the company’s books. Our inventories were a known point of weakness, but you try to manually inspect thousands of racked servers – we’re growing too fast to do this by sneaker-net. Data storage and the networks were even more of a challenge.

The problem was that we have literally hundreds of business and productivity applications running throughout the enterprise, a surprising number of them squirreled away in business departments, hosted out of wiring closets, and even several under developer’s desks. We also discovered more than a few applications that were un-recognized (by the I.T. group), including several that were hosted remotely, some of them left over from past acquisitions and mergers.

Application Mapping Issues We knew that a key set of inter-related tasks in DCR planning is the discovery, analysis and documentation of the literally thousands of real and suspected application inter-dependencies that exist among the applications that run the business. We also knew that few business applications stand-alone, and the ways that they can interconnect and share data are as varied and imaginative as the developers who created and maintain them. But, let’s be honest: we were quite aware that the task of identifying, verifying, and documenting application inter-dependencies in a mid-sized to large organization probably isn’t going to happen. For us, the job was far too big, and anyway, the application interrelationships keep evolving too fast.


15/71


But this was a critical area of trouble for our project because in a DCR, you do have to eventually shutdown, unplug, and restart real systems with critical applications that users must depend on. Oh, we did the virtualization thing. It’s great, but not yet a silver bullet. With proper DCR planning, we could have identified the major problem areas, and developed a better (welltested) risk management solution. But, we didn’t do as good a job as we thought we did.

Root Causes As the extent of the problems triggered by the relocation became large enough to impact our business users and key customers, we couldn’t keep hiding application performance issues, data losses, operational inefficiencies, service interruptions, and rising costs. Our business executives started asking pointed questions and they demanded real answers. Then, they turned to an independent DCR company for a post-project forensic analysis – something that the lawyer for a Board member suggested.

Data Center Relocation 101 So, you’re a very capable manager in the IT department of a medium-to-large corporation. You’ve just come out of an executive planning session where you’ve been told that the company needs to relocate its entire IT operation to a newer and larger facility in another location. Your data center is growing at twenty-five percent annually and you are nearly out of room. The new financial model says that this relocation has to happen within the next eighteen months, and sooner is better. You’ve also found out that although this move is considered business-critical, it doesn’t mean that I.T. will be suspending any other major initiatives, so the project is just additional workload and responsibility for your already over-worked staff. Oh, and one more thing — the entire project belongs to YOU!

Our CIO spent many weeks squirming on the C-Level hot-seat while the DCR consultants were engaged in some pretty intensive forensic analysis. We spent more time and money, and unfortunately several technical managers and staff, and a few project managers were asked to move on. What ultimately emerged was a pattern of “failure by neglectful competence” – that’s what the forensic consultants called it. We were simply too good, too confident at running a data center to recognize that the DCR planning process required for data centers of our size and complexity was sufficiently unlike any projects we had experience with. It almost guaranteed serious problems. The project was, and remains, a very painful and largely unnecessary lesson. We also have a new CIO.”


16/71

It’s YOUR Party …Now! Sounds unlikely? Hardly.

You might even know how to do some of the actual work.

It happens all the time. Data centers get moved and consolidated for a variety of business-driven reasons, such as mergers, acquisitions, or divestitures, running out of room to grow, antiquated physical facilities, regulatory mandates, or inadequate utilities access.

Still, it is unlikely that you will know everything required for a successful project - how many nails, boards, shingles, parts and pieces, or man-hours the whole effort will require, or even how long it can take the foundation to set in your weather.

But whatever the trigger, it usually comes down to a simple arithmetic — cost control. Even when your data center runs out of room, there is a business analysis that constantly runs between expanding the current site and finding a larger facility. Considering that cost is always a key factor, it isn’t at all unusual for someone in senior management to decide that the entire DCR project must be accomplished with existing resources, minimal added expense, and on an accelerated schedule. You know the drill.

And you also must understand the building codes and other local requirements involved. These are among the hundreds of critical details that the building contractor must know, understand, and be ultimately responsible for. The experts all say that if you want to make sure that you aren’t taken advantage of when building your new home; it doesn’t hurt to do a little research ahead of time. If you’re building your dream home, read up on home construction techniques, financing options, and basic construction project management.

How Does a DCR Happen? OK, it’s now your project. What will you do? How are you going to move the workload? What will it cost? What is the impact on your staff? How do you manage the project? How do you do all this and still go home at a reasonable hour? Or, do you? While you may have some ideas, if you’ve never been involved in a major data center move before, there is a lot of information that you are probably missing right about now. From a project perspective, a DCR project is a bit like building a large house – think of something that your favorite celebrity would live in! You have some general knowledge of how it is supposed to be done: grading and preparation, foundation, walls, roof, interior, flooring, HVAC, electrical, plumbing, landscaping, building codes, financing, etc.

If you’re responsible for a data center relocation project, you have already started that process by reading this Guide Once you have the core team in place, you can begin to develop a detailed scope: • • • • • • • • •

Is this relocation within the existing campus or to a new facility at a distance? What will move? What will remain? When is this supposed to happen? What is the financial plan? Will you do it with in-house staff? Will you use consultants? What migration strategy – forklift, swing, swap…? Who are the key stakeholders?

When your project’s scope has been defined, the next step is to begin a high level analysis of the major domains to be moved or consolidated.


17/71

1 23 Migration Domains A “domain” can be the server farms – or just one of them. Each “domain” will have specific business, operational, technical, and financial characteristics that must be identified, analyzed, and documented at a high level. This is where the DCR “pre-planning” assessment pays for itself almost instantly. Expect it to take anywhere from a few weeks to several months, depending on the complexity of your data center environment(s), but at the end, you will have a clear (if high-level) map of the entire DCR project. As you work through the assessment process, DCR subject matter experts can provide invaluable assistance – reducing the complexities, time, cost, and effort required to understand your data center environment(s) at the level of detail necessary to begin the next, most important phase – the DCR Planning cycle.

Do the Assessment It really can make a difference. The DCR Assessment is an effective, lower-cost requirements assessment and validation service that is designed to transfer knowledge about how DCR projects are organized and planned. Typically, it usually takes a few weeks, not months. It will actually cost less than an internal effort because the DCR planning SMEs already know what, where, how, and why to find the right information. And, it will prepare your team for the real work ahead.

Relocation or Consolidation? From a practical standpoint, there are two key differences: •

Relocations “MOVE” people, equipment, applications, and data from one physical location to another. Usually, the distance is more than a few miles.

•

Consolidations “REDUCE” the number of discrete systems within an existing data center. A consolidation is also normally done in concert with a relocation to cut costs and simplify the migration.

Relocations are generally focused on location changes and may include transformational improvements in the data center. When you are committing the large capital expenditures that a significant DCR project requires, your business and I.T. executives will also want to see measurable improvements in customer satisfaction, reliability, operational performance, and especially ongoing cost reduction. Consolidations, while under much of the same productivity improvement pressure found in a relocation project, are more often tightly focused on immediate operational cost reduction objectives. Reducing the number of active systems and applications required to support the enterprise can yield substantial licensing fees and support savings over time.


18/71

DCR & Virtualization Where the benefits of cost reduction are most readily observed is in the use of “virtualization” solutions that enable systems administrators to load and run multiple applications on a single “under-utilized” server. That solution can save money and increase system productivity quickly. Virtualization as a DCR implementation tool is also extremely useful in the relocation process because it can be used to move applications from “leave-behind” source systems to target systems that have been pre-installed in the new data center. With the right business approach to application virtualization, you can “move” the applications and data, not always the hardware. While virtualization has sometimes been accused of being “oversold as a solution for everything from dandruff to aging,” it has become an extremely valuable and important DCR tool, and one that any DCR planner should be conversant with. It is also widely employed in the DR/COB environment for its obvious benefits. But it won’t help that much if the detailed planning isn’t there. Virtualization tools such as VMware, Virtual Iron, Microsoft’s Virtual Server, and Xen can provide such benefits as: • • • • • •

Server consolidation and improved resource utilization Reduced hardware, software, services and facilities costs Improved operational efficiencies, with greater scalability and adaptability to change Faster application deployment Improved security Enhanced DR/COB environments

DCR Planner Trends Government regulations may impact your DCR project; one of the many things you need to analyze and plan for. After 9/11, Federal regulations mandated that certain financial services industry firms (and others) establish additional “primary/backup” data centers outside the metropolitan areas of the major cities. In some cases, a minimum distance of 200+ miles between data centers was mandated. Therefore, you may need to check with your legal department to see if there are any facility location requirements for your company or government agency. Intended to enable fast recovery in the event of a terrorist attack or natural disaster, the mandate has triggered lots of new enterpriselevel data center construction or relocation projects globally. Because this process is still getting underway, there are a relatively large number of very complex data center relocation and consolidation initiatives presently in progress. That fact has resulted in a significant tightening in the availability of top-flight DCR planners worldwide. This trend also means that a number of people are entering the field, with limited real-world experience and technical skills.

Reduced power, cooling and floor space also help reduce operational costs and improve maintainability of complex systems. A future PMCA Guide for Consolidation Planning will cover virtualization and its role in consolidation planning in greater detail.


19/71

The limited availability of capable DCR planning talent is expected to continue for some considerable time (years) as major companies and government agencies continue to transform their ever-more complex I.T. operations and infrastructures in response to increased global competition and political realities worldwide. Outsourcing and consulting firms are also having increased difficulty finding “Top 100” people for the simple reason that it takes years to gain the experience and insight to do data center relocation and consolidation planning well. So, plan well ahead for your DCR project. Another trend in I.T. with regard to many projects is to keep them in-house where possible. While that is clearly a preferred solution for those organizations that recognize and support the business value of their own I.T. capabilities, planning a DCR requires skills that may not be widely found internally. Even some of the largest corporations and government agencies, with a wealth of superbly talented people, have determined that, for complex DCR project planning, the most cost effective solution is to integrate specialist DCR planners and other SMEs within the core project team.

Getting Started Right OK, you have the ball. We’ve covered the basics of DCR “pre-planning.” So, how do you get started? What do you do next? Who do you need to contact first? Here are some suggestions:

Understand the DCR Business Case Executive management didn’t decide to invest in a data center relocation initiative without a sound business case. Get it. Read it. The DCR business case and financial analysis documents will serve as a baseline for future business decisions. Use them to make sure that the choices you make support the original goal. On a large relocation, it is possible to get so wrapped up in the effort that you forget why you are doing it in the first place. Keep your business focus and revalidate your actions and financial position as you proceed.

Identify the Key Stakeholders Your executive sponsor is one. Business unit leaders, major application owners, I.T. operations management… the list of key DCR stakeholders is going to get quite large, very fast. Managing them will be another of your many full-time jobs. Stakeholders can include people not usually associated with an I.T. project. Because of the scale and enterprise-wide impacts of a DCR, you are going to have to be careful to identify the people and groups within (and external to) the organization.


20/71

Assign a Project Manager Hopefully, yours will have some DCR experience. Even for a small relocation, there is a need for a full time project manager. The perfect candidate for the position will have a good overall knowledge of the IT department and operation. If at all possible, the DCR project manager should NOT be part of an existing technical support team (Mainframe, Server, Network, Applications, etc.) because this can sometimes lead to a bias and over - concentration on a single area. More critically, selecting such a project manager can also mean that you really don’t “have that person on the team.” Prior responsibilities may demand his or her attention in ways and at times that will place the project at-risk. Basically – don’t “share” your DCR PM with anyone else!

The next best thing is to talk with a number of DCR specialist firms. Ask them questions about the types of project risks they most commonly see. Understand that consultant firms are going to be reluctant to sit at the end of a phone while you pump them for lots of details. If you want the benefit of their experience (and you really do…!) be prepared to engage them.

Implement a DCR Methodology Does this mean that there is more than one DCR methodology? Yes, there are probably almost as many ways to conduct a data center relocation or consolidation as there are consultants and I.T. vendors who offer these complex services.

The person selected as the DCR Project Manager should be able to work with and manage senior executives and technical staff, vendors, consultants, DCR specialists, services providers (utilities), site construction PMs, and even local government agencies if necessary.

However, they all have some basic commonalities. Almost everyone now follows the Project Management Institute’s (PMI) project management process for good reason: it works – and it’s the place to start organizing your DCR methodology.

S/he should obviously be detail-oriented and an effective team leader, but also able to focus on and communicate the overall project objectives and strategy to stakeholders and teams.

The PMI methodology is powerful and very well-documented, and there are many very capable project managers who are trained in the PMI approach. An increasing number even have the prestigious Project Management Professional (PMP) certification.

Do Some High-Level DCR Research Talk to your vendors and peers. Find out who has done a similar relocation recently and talk to them about what they learned.

However, don’t confuse an effective DCR planning methodology with a Project Management methodology. They are complementary, but different enough in a number of subtle ways as to contribute to problems later on if you aren’t aware of the differences in focus and technique.

You will find people who have gone through this and more than willing to share their experiences; some of them may even share both good and bad lessons learned. This can give you valuable insight into what you will be facing. Of course, you’re quickly going to find that nobody is really willing to candidly discuss their DCR project failures – and that’s where you’re going to find out what you really want.


21/71

These details will be uncovered over time, but only with a lot of hard, gritty work by you and the entire DCR team. Learn the process (to be expanded on later) at the start and it will help you manage workloads and stakeholder expectations in the future. A good methodology is based on many years of real-world data center relocation and consolidation experience. Use it because it is a well-proven and cost effective approach.

DCR Methodologies Project scale is the first differentiator. Outside of the largest organizations, those with mature Program and Project Management Offices, the requisite skills and experience to manage a project of this size and complexity may be scarce. Complexity is another differentiator. Unless your PM team is DCR-experienced, the amount, diversity, and depth of detailed information that must be uncovered, analyzed, and acted upon will be a real challenge. Another key differentiator is focus. In many small systems and non-I.T. business or office relocations, the focus is more on “when we’ll complete the project.” How it is accomplished just isn’t that difficult. In the DCR setting, “when” is of course, important. But even more critical is “how” we plan to get there. “How” IS difficult. The focus of your DCR must be on how your systems, applications, and networks will migrate. In a business environment, HOW you execute a mission critical project, such as a data center relocation or consolidation, determines your true schedule and costs. “When” is merely a derivative of that function. As you begin to plan your project, you will quickly notice that no one has all the details — not you, not the data center operations and technical staff, nor your vendors and consultants.

Create a DCR Project Communication Plan As soon as you know that you are going to relocate your data center, start working on your communications plan. Generally, your plan needs to document how, when, and what you will disclose to both internal groups and external customers. At a minimum, you should start your internal communication activities early. This allows you to tap into the knowledge and expertise of your technical staff and application owners, and will help you uncover those project-fatal “gotchas” before they “getcha.” Also keep in mind that one of the best tools for communicating about a project like this is an online discussion forum and a repository for documents and project plans. If properly structured and maintained, it can be a central place where people can ask questions, get the right answers, and find the latest documentation on the project. PMCA prefers to use Microsoft’s SharePoint and Project 2003 or later for this.

PMCA has provided a DCR Communications Plan template as part of this Guide service. You can order the free DCR Template Kit at:

[email protected]


22/71

PMCA’s DCR Template Kit

Use Templates More Effectively

The DCR Pre-Planning templates that we offer can help you define the scope and key areas of focus for your project.

When you contact us, we would like to ask a few basic questions about your DCR project and offer our cost effective DCR Assessment service.

Customize them for your needs, but don’t confuse “pre-planning” with the real work that lies ahead. To do so, simply increases the risk of downstream failure. And nobody wants that. PMCA’s DCR Project Pre-Planning Guide Templates: Available Now Pre-Planning Communication Plan Template (Excel) Pre-Planning Project Initiation Presentation (PowerPoint) Pre-Planning Issues Assessment Template (Excel) Pre-Planning Inventory Assessment Templates (2) (Excel) Pre-Planning DR/COB Assessment Template (Excel) Pre-Planning Organizational Matrix Template (Excel) DCR “Pre-Planning” Task-List Template (Excel, Project) Each template has instructions on how to use it and what role it plays in the pre-planning cycle.

To get the FREE PMCA DCR Project PrePlanning Guide templates, simply register at www.pmca-us.com with your name, title, corporate email, and phone number. Or, call us at:

1.954.962.9864 Register FAST by sending us an email to: [email protected] Other PMCA DCR Project Planning Guides are being developed. Visit our Web site frequently to find out when they will be available.

Once we talk with you about your requirements, the template kit will be sent to you at no cost or obligation. We will also follow-up later to ask if there are further ways we can help you succeed. DCR planning in a large data center environment is really not a “do-it-yourself” activity – even for the experienced project manager and technical groups. Not, at least, for a project of any large size. There is much more to planning a successful DCR project than reading a few white papers or filling out nice templates - however helpful they may be at the beginning.

Note that we do not provide a “DCR project plan” in the Template Kit. The reason for this is that every DCR has common characteristics and tasks, but these are well known to project managers. The really difficult task definitions will vary widely from project to project and require a DCR subject matter expert to identify and research them. We DO, however, provide a “pre-planning” task list in MS Excel and Project 2003 format in the DCR Template Kit. If you intend to excel at your DCR project, (and we know that you do) informed, careful preplanning is the best way to prepare for the real work ahead – the detailed discovery, analysis, and planning that will be required before the actual move dates occur. PMCA has experienced DCR planning specialists in all the key industries, experts who can save your team a lot of time, money, and pain. But, you do have to engage us if we are to be of optimal benefit to you.


23/71

DCR Risk Awareness Planning Risk of project failure is always present in any I.T. initiative, as it is in virtually every area of human endeavor. Industry data has long suggested that as many as 80% of all I.T. projects fail to deliver the benefits promised. For a DCR project (because it IS missioncritical!), failure can mean anything from minimal disruption and rapid recovery to substantial business interruption, loss of revenue, damage to your firm’s reputation, civil lawsuits, and even government investigations. All of these are nice things to avoid.

Risk Identification Many (but, not all) project risks can be mitigated or even avoided - only if they are identified and planned for in advance. For a DCR project, there are a few key areas of risk that always demand closer scrutiny: • • • • • • • •

DCR Forensics Forensics is the art of determining “root causes” of project failure through close examination of events and decisions leading up to that result. Think of it as “CSI for DCR.” Because too many DCR projects neglect “pre-planning” phases to “save money,” practice is growing fast. You should everything you can to avoid the need for type of post-mortem outcome.

the this do this

This Guide is a way to help you do that. But, should your project require it, PMCA also offers a DCR Forensics service – to research and identify the root causes of data center relocation and consolidation project failures. The key, again and again, to DCR project success, is an abundance of detailed planning. And a very good DR/COB plan.

Project management and execution Budget Technology & vendors I.T. management, process, staff Capacity Planning Facilities & infrastructure, security DCR planning & risk DR/COB plans

There are always a few too many companies that fail to learn from the DCR mistakes (and successes!) of others.


24/71

The DCR Pre-Planning “Trap” Avoiding DCR Project Hell Generally, the fastest, most effective way to get yourself trapped in a career-damaging “DCR project from Hell” is to assume that because your I.T. group “operates the data center, they can move it.” Yes, the I.T. group does an incredible job and under truly daunting pressures. That’s EXACTLY why they are the worst people to ask to move the data center. They have NO time. While they are the best sources for much of the detailed information you’ll need, getting it will be a constant source of aggravation. The second-best way is to grab a bunch of stuff off the Web – or use something provided free by any vendor (including PMCA) – and rush to “do-it-yourself.” On the dubious premise that somehow you’ll save money. You won’t. But, if/when you decide that you really don’t need the DCR specialists, we’d like to point you back to the case study at the beginning of this Guide. The key to a successful DCR is simple: planning, more planning, checking your planning techniques, validating the plans and planning methodologies, sharing them with others to find the gaps, and re-checking everything... again and again throughout the life of the project.

The problem that many organizations find when they begin to plan for a data center move or consolidation is that they don’t have the time to do the pre-planning well. Or, they don’t take time because “it’s not really needed” for this project. A modern I.T. organization is extremely busy. DCR planning requires time and effort that must be taken away from day-to-day operational and development needs. You can fall into the trap of thinking that it’s not really that necessary to understand the interlinkages among applications and systems to the depth required to pull them apart, move them, and reassemble everything so that it works as before. The levels of detail often required for a successful DCR are large – and routinely underestimated. Since DCR project failures can (and do) put companies at significant risk of real economic loss and business failure, management, shareholders, and investors (and their legal counsel) will want to know what happened and why (and who to blame…). Don’t give them the opportunity. The key, again, is pre-plan, plan, validate, and then move or consolidate.


25/71

Ten Key “Givens” For a Data Center Relocation Project Before you start the relocation planning process, there are some things that you need to keep in mind during the life of the project. It seems that, no matter the circumstances, there are certain “givens” that apply to any data center relocation. Everyone runs into these and so will you. But, armed with advance knowledge of what to expect, you will be better equipped to handle these challenges.

1 23

Pre-Planning Planning Move I.T.

These “givens” are: 1.

Relocation is a business issue, not a technical problem 2. Relocation is STILL & always a business issue, never a technical problem 3. You can never over-plan the project. But, far too many teams under-plan theirs’. 4. There is no such thing as too much communication. But, doing it right can make a BIG difference 5. Procurement and logistics are always a problem –with few short-term solutions 6. Construction projects never finish on time. And, the Certificate of Occupancy WILL be delayed 7. Network providers will always miss at least one critical cutover date, maybe more. 8. It’s much, MUCH harder to move people than equipment 9. Something will go wrong. And you probably won’t know it until the pain reaches a disruptive threshold 10. Take care of your people, communicate with the stakeholders, order late-night pizza frequently, and try to make it fun Let’s examine each of these in a little more detail…


26/71

Relocation is a Business Issue Notice how this “given” occupies both the first and second spots on the list? It’s that important. From a purely “technical” perspective, DCR projects are pretty straightforward. Figure out what goes, what stays, and then plan for when and how it all happens. Easy…, right? From the business perspective, it is a much more complicated problem, especially when you consider cost, risk, and customer service level agreements. In a perfect world, there is only one way to relocate a data center. This isn’t quite a perfect world. From the technical perspective, you could start by replicating all of the equipment, data, and applications in the target site, test everything fully, and then migrate the entire workload and data to the mirrored site. It’s the least risky approach, with a built-in back out and disaster recovery capability. Unfortunately, it’s so incredibly expensive that very few companies can afford it.

You Can’t Over-Plan the Project Or, over-research it. (But, somehow you still can’t seem to get all the information that you need…!) It seems obvious, doesn’t it? Relocations, by their very nature, are highly complex. The further you can drill down into the details of a system, application, or infrastructure, the less likely you are to miss something critical. The problem here is that, in almost every case, the systems that you need details on are more complicated and less-well documented than they look on the surface. Also, the people who are most knowledgeable about the detailed information that you need have other jobs. The work of a data center doesn’t stop just because you’re going to move it. They are not necessarily going to want to add to their own workload to stop and give you all the details you may need. Not without a clearly defined reason and lots of “up-close” management support.

The point here is that very early on in your project; you will be faced with the need to balance the best technical solutions against the needs of the business. Almost always, it comes down to cost/benefit analysis.

Worse yet, if overly pressured, they may just give you a high-level task list without really thinking it through — just to make you go away.

In any company that is concerned with the bottom line (pretty much all of them!), you can expect your C-level business executives to make the tough decisions based on the business case and current budget analysis.

Planning can yield the critical details needed to catch a problem before it becomes a disaster. The key is getting the details you need without losing the support of your I.T. groups and business users.

The technical staff rarely makes the truly important DCR decisions (the expensive ones); although a smart CEO listens to his or her CIO very carefully. Your job is to be sure that the decision-makers are kept well-informed on the key DCR decisions that must be made. Relocation is always a business issue. And it’s ALWAYS about the money. But, that said – while everyone wants to have a DCR project that is “better, faster, cheaper…,” you know that you can pick only two of the three options. Choose wisely.


27/71

There’s No Such Thing As Too Much Communication As stated before, relocations are complex. How complex? More than you can probably guess right now. One way to keep from missing a multitude of critical details is to share your plans with as much of the staff and customer base as possible. No one can know everything, but the more people who (actually) look at your plans, the better your chances of a smooth deployment. Inevitably, someone looking at the project from a different perspective is going to catch something that the relocation team missed or hadn’t considered. Communication is especially important in relocations where some or all of the employees are moving. In general, people dislike change. The sooner you let them know what’s going on, the easier the transition on everyone involved. When moving people, get them involved in their new workspaces; their managers will almost always be willing to help decide who goes where. And, maybe even let them determine where the office ferns and other plants will get placed. So, while there is no such thing communication, it can be done that doesn’t help the cause. If problems, get them addressed possible.

as too much in a manner you run into as soon as

Procurement & Logistics Are Always a Problem There is always Procurement (purchasing the stuff you need), or Logistics (getting it where you need it) activity associated with a data center relocation — equipment, services, or both. The larger the relocation effort, the more your procurement and logistics systems will be taxed. If you happen to be constructing a new computer room or a whole new site, you are looking at hundreds of new items. Why is this a problem? Even at the smallest scale, most companies over-estimate the efficiency of their procurement and logistics systems. The four weeks that it should take to get a critical component soon becomes six, seven, eight… and your schedule will be jeopardized. In any reasonably large DCR, you can expect things to break or go missing. And just as dependably, your spare parts inventory will probably not have what you need. No problem, we’ll get it next week. That’s not the right answer if the part you need is the one that’s keeping the mainframe from being installed by Monday morning. Think it won’t happen? It does, all the time. That’s why in a major DCR, the key vendors may urge you to pre-stock critical components. Some companies have even placed a corporate jet at the DCR team’s disposal – with a standing weekend order at the nearest IBM parts depot. Plan for the worst; expect nothing less than the best. Using the services of a professional high technology moving and transport company in your DCR logistics plan can help reduce the risks inherent in moving expensive assets.


28/71

Construction Projects (Almost) NEVER Finish On Schedule If your relocation depends on the construction of a new building or computer room, be forewarned! Construction projects (almost) never finish on time, and even when they are close to the schedule, there are always punch list items (follow-up work) that won’t be completed in time. That is just the way it is. Weather, materials and shipping, labor strife, regulations, architectural re-design (fixing mistakes not found earlier), money… these are just a few of the more common reasons for large-site construction project delays Knowing this fact ahead of time, your job is to build some contingency time into the schedule, and then make sure that the contractors will let you into the building as scheduled, even if they are having delays. And you’ll need to coordinate with more than just the construction boss (PM). Other vendors can lag behind for many reasons; schedule slips have a way of cascading. A one week delay in March could mean three or four weeks in August. And your move-in date is midSeptember. Working with the construction PM, you can plan to pre-install key “virtualization” systems, sometimes well before the actual move dates. Of particular importance are the rules around the Certificate of Occupancy. Rarely will the local government’s building inspector grant “resident access” to a working construction site – not until the CO is approved. This means that your team may be allowed to go in and install equipment once the electrical, HVAC, fire suppression, and other trades are finished with their work. But you won’t be allowed to stay and work onsite. Therefore, any systems that you are able to pre-install must be remotely manageable. That could be tough if the network vendors haven’t yet completed their work.

Again, while “planning” matters, ”pre-planning” really matters. Often, the best approach here is to focus on auditing and tweaking the details of your DCR plan while the construction crews continue prepping the site. Trying to install racks while the contractor is still painting walls and installing trim is probably not going to work out that well.

Network Providers Will Always Miss at least One Cutover Date If you have a large network with a lot of circuits to migrate, this “given” definitely applies to you. No matter who the provider is, sooner or later they are going to miss a key circuit cut. In order to keep this from being a problem; never make your circuit cuts at the same times that you are relocating workload. Instead, schedule them as independent activities, with the ability to fall back to the original configuration if the circuit cut fails. We all know that managing data, voice, and video utility installations can be challenging. Working closely with the vendors can help reduce the stress and possibly even improve the schedule a bit.


29/71

Some Things WILL Go Wrong In any relocation project, this is inevitable. Something, somewhere, is not going to go as planned. How do you deal with this? First, mitigate the chances of something major happening by planning the project with as much detail as possible (sound familiar?).

It’s Much, MUCH Harder to Move People Than Equipment Generally, when moving equipment and large groups of people, you will spend far more time and energy on the people part of your move. The reason for this is simple — equipment doesn't talk back or use delaying tactics. In a people move, you are always going to end up making someone unhappy.

Next, build risk contingency plans for the things that are most likely to go wrong or have the greatest impact on your relocation (like construction being late). At the heart of any DCR project is the Risk Plan. Here, you work to identify, assess, and develop mitigation strategies for each significant risk that the project may encounter. This is, of course, easier to say than do.

Perhaps the new site is farther from home, or they don't like the new cubicles. Maybe the great restaurants near your present location are far from the new one. If a person is upset about moving for any particular reason, they will quickly find validation for their negative feelings.

But, Many Things Will Go Right!

In a sufficiently large group of people, this can quickly become a nightmare. The easiest way to work around this is with good early and continuous stakeholder communication.

The satisfactions gained from surmounting incredible challenges and handling problems that would stun lesser souls - these are pleasures earned through hard battle against formidable hurdles.

The key here is – the “real” stakeholders aren’t just the ones in the corner offices. Sometimes, it also helps to make a dissenter part of the core team. Once you get them directly involved in the decision-making process, they are more likely to support the project and make your move that much easier.

As the pressure mounts, it will often be hard to take the time to find the “fun” in a data center relocation project. But, it’s there. Find it.

It’s difficult to over-estimate the complexities found in a major DCR. In fact, it’s dangerous to do so. But remember, some amazingly large and stunningly complicated data centers have been moved or consolidated over the past two decades. Yours is just the next one to come online.


30/71

Team building events, an afternoon off when it’s slow or catering lunch once a month, can make a huge difference. When you get to the actual cutover activity, it may last anywhere from one night to many back-to-back weekends. During this time, one of the more effective morale-boosting things you can do is keep everyone well fed. Once again, this seems like basic common sense, but you would be amazed how fast things fall apart if people get hungry and cranky. Most IT professionals are lucky if they get to do one major relocation project during their career. The question is how they approach it. If it is nothing but stress and unwanted work, their (and your) likelihood of success rapidly diminishes. At the close of the project, when the postmortems are finished, don’t forget the certificates, bonuses, and vacation days accrued. By this time, your teams will probably be exhausted. So, if the project’s size and duration warrant it, a little executive recognition, in the form of a team dinner meeting and awards presentation, works miracles. It’s money well-spent. Approach your DCR project as a unique opportunity to do something completely different — have fun with it. OK, at least you can look back on a really tough job that was executed well…

DCR PrePlanning Checklist You’re now ready to begin. First, here is a high level checklist to help you get better organized for the critical DCR Pre-Planning phase. Pre-Planning means those key activities and dependent tasks necessary before you begin the arduous process of detailed project planning. You’re setting the stage here to ensure later project success, so resist the temptation to skip this phase. Start by putting these tasks into a project plan:

Business Case Communication Plan Risk Assessment Financial Analysis & Management Key Stakeholders o Business Units o I.T. o Customers o Other Legal, Regulatory Contacts Project Sponsor Project Manager Facilities Manager o Current Site(s) o New Site(s) Key I.T. Managers o Mainframe/Host o Servers o DASD & Storage o Printers o Data/LANs o Voice o Other Vendor Contacts o Hardware o Software o Services Key Inventories o Hardware o Applications o Networks o Other DR/COB & Security Utilities Other


31/71

DCR Project Management Checklist While a detailed coverage of DCR project management practices and methodologies is outside the scope of this Guide, this checklist will help focus your attention on the importance of relying on accepted PM standards. • • • • • • • • • •

DCR Project Lifecycle Project Integration Management Time Management Cost Management Quality Management Human Resources Management Communication Management Risk Management Procurement Management Vendor Management

The size of your DCR project will determine the structure of the PM team. For a large enterprise initiative – or one that will have global activity - you should implement the PMI Program Office model: an overall program manager will coordinate the activities of project managers. Project managers should be assigned to each key domain and/or location (in multi-site DCR’s). Domains might include: • • • • • • • •

Mainframe/Host complexes Server farms DASD and Storage systems Specialized equipment Outsourced systems and services Procurement and Logistics Facilities Voice/Data, etc.

1 23 “DCR, …it’s what we do.” PMCA is an independent trusted advisor to Fortune 500 and government clients worldwide. PMCA offers complete project management services, including consolidation, migration, and relocation planning - for your datacenters and other IT infrastructure projects. PMCA data center relocation planning services are used by the Fortune 100, government, and leading I.T. consulting firms, because we are globally recognized DCR specialists.

www.pmca-us.com

The all-too-common practice of assigning one PM to run the entire project may work for small efforts. But, a large DCR will need a dedicated PM team.


32/71

The Data Center Relocation Process Like most things in life, there is a process associated with data center relocations. The optimal process is composed of five major phases. Let’s take a look at them in their most basic form: What is important to remember is that the bulk of the work is in the planning. In fact, there are really two plans. Obviously, there is the plan for the actual relocation activity — the move or moves. This is referred to as a cutover plan.

The other is the plan for creating the cutover plan(s). This is the part that takes the most time and effort, and is what you will spend most of your time managing.

How Does It Work? In its simplest form, we can compare a data center relocation to moving from one house to another. You start by figuring out WHAT needs to move, what needs to be replaced, and what needs to be left behind. This is a real opportunity to set up your data center (household) exactly as you want it, so this is the beginning of a series of business decisions that need to be made.

Graphic available in Template Kit.


33/71

Once you figure out what is moving, then you can break it down to simple categories. In your home it might be by room, kitchen, bedroom, basement, etc.

Unfortunately, most people don't have an unlimited supply of money, and with a large data center relocation project on the table, neither do most businesses.

In your data center it might be mainframe, DASD, tape, and storage systems, application servers, networks, printers, etc.

As in a personal move, there are trade-offs that must be made. Every decision is a constant balancing act between cost and benefit – remember that business case?

If the move is spread over a number of nights and weekends, these groupings become even more well-defined, and it becomes more important to move certain categories together in “move groups.”

This is the point where prior DCR experience is now key. If you have not yet reached the point where you understand the value of using DCR planning specialists, you’re about to.

Moving the kid’s bedroom and leaving their bathroom stuff behind is not a good idea. In a data center relocation, we call these “move groups” or categories a “complex.” They operate as whole; you move them the same way.

The decisions on what goes or remains behind that are made by line-of-business and I.T. executives – who MUST be involved at this stage - can have both short and long term business, operational, financial, and even legal impacts within the organization.

Next, we decide HOW to move the belongings. Either at home or in the data center some items are going to require special attention.

At this point, you should be assembling an organizational matrix of stakeholders, business groupings, and the key I.T. resources that you will rely upon to get the work done.

The Baby Grand Piano from your Grandmother and the server farms both need extra attention. However, the dining room table and chairs, like some of your old servers, are too outdated to move so you will just buy new ones and have them delivered ahead of time.

Make it a high-level effort for now, digging for the details will come later. We have a template for that too: Pre-Planning Organizational Matrix

Notice that this is usually a function of time and distance. The more time you have, and the closer it is, the more stuff you are likely to move. If you are moving a great distance and you only have one day to do it, then a lot of items will have to be purchased. Obviously, cost is a factor. The simplest solution is to buy everything new and only move the things you absolutely have to — like the dogs, the kids, and the really critical things such as specialized printers, mail sorters, and other items unique to your organization or industry.


34/71

Plans vs. Schedules What is a plan? schedule?

How is it different from a

In this document, any time we refer to a "plan," we are talking about a document set that contains information about what, where, and how things will be done (including the "schedule"). The plan also includes additional reference material, such as resource and vendor contact lists. The "schedule" is a document that lists the specific tasks that need to be completed, when they are due, and who will do them. Confusingly, people sometimes refer to this “schedule” as the "plan," when in fact it is only a small part of the "plan" - and apart from the “plan,” the schedule is pretty worthless for explaining how, where, or why something must be done at this particular time. In any project as large as relocating a data center, building task flexibility into your plan is a very good thing. Avoiding the “do-it-yourself” approach, at least in key areas, is another good practice. Now that you have completed the what, how, and when phases of the relocation planning process, you can start working on the details and doing the detailed planning work. Again, let’s use the “new home” analogy. Tasks at this stage can include reserving a truck, hiring the piano movers, buying boxes, and packing materials. You will need to carefully coordinate transportation options, considering weight restrictions, special vehicles for delicate equipment, and staging areas to sequentially arrange how the equipment must be removed from the old facility and arrive at the new destination. Special crating requirements also need to be planned. Planning includes creating a timeline (the “schedule”) for the actual move periods. Short or simple moves require less complex timelines.

On the other hand, if you only have six hours to make a major move, it is imperative that everybody knows exactly who's doing what, how, where, why, and when. This phase is the longest in a data center relocation project. It takes a lot of time to make all your procurement arrangements and work out your detailed timelines, which are normally at the hour-by-hour time window detail level.

Move Group Plans – Then Schedules Creating specific move groups is a major challenge and well beyond the scope of this “pre-planning” guide. But, once the DCR plan reaches a working “final” draft stage and you have completed your move groups, it is time to determine WHEN you want to move – the schedule. Part of this is driven by how we decided to do the move. Right up front, you should resist the pressure to move everything quickly. It won’t come from the I.T. group, but from rather from business users. The best defense against such pressure is to have and follow a well-designed Move Group Plan. That plan will be one of the more challenging tasks that you and the DCR specialists must develop during the detailed planning phase of a DCR. The Move Group is simply a very detailed list of ALL of the systems and their dependent components, applications, and everything else that depends on members of the Move Group that belong to a particular domain.


35/71

This domain could be an application (Payroll, HR, ERP, CRM, etc), or a business department, a remote location (branch office’s I.T. center), or other categorization specific to your company.

Security Nobody likes to think about bad things happening to good projects. But in the inevitable chaos of a large move, things can walk off. It would not be a good time to discover that the devices that hold your company’s payroll or customer data have been misplaced. Location, equipment, personnel, and transit security must be planned well in advance of the need.

The Move Group Plan should also contain detailed information on who needs to be onsite during the equipment relocations, what procedures are required to boot up re-installed systems for test runs, who to call when things go bad, and how to order late-night pizza, coffee, or a paramedic. Which brings us to an important topic…

Whether you use your organization’s internal (contracted) security service, or hire someone for the job, a carefully thought out plan is essential.

Move Groups & Schedules

Emergency Services

A few years ago, an unusual DCR project moved an entire airborne scientific laboratory as a set of tightly inter-linked move groups from an older aircraft to its new replacement.

Many DCR migrations and consolidations happen at night or on weekends. You will be working around heavy equipment and lots of activity. There will be the occasional cuts & bruises, and hopefully nothing more. Have a good First Aid kit onsite.

The team did this while both planes were on a Navy aircraft carrier at sea. And they did it all in one “96-hour day,” as the team leader described the effort.

But, in the event of a serious health problem, who are you going to call? 911 may be the obvious choice, but if you call from a cell phone (because the phone company hasn’t yet finished their work), be prepared to give EXPLICIT directions to your location. Know where the local hospital and emergency services are in relation to your new location. Remember, if you are moving to a newlyconstructed facility, the location particulars may not yet be updated in the emergency dispatch center’s geographic mapping system.

While your project’s move day may not be quite so dramatic, determining when and how to move is one of the major tasks that you, as the DCR team leader, must complete. If you decided to do a household move all at once (the “Big Bang”), and in one large truck (or a fleet of them), it may take an entire weekend. That's great, but if it turns out that there is only one weekend that your friends are able to help out, your schedule will be tied to their schedule. Like it or not your move weekend has just been cast in concrete.


36/71

Schedule vs. Freeze Dates There are specific dates when nothing in I.T. moves: end of month, end of quarter, holidays, full moons, etc. Your organization maintains such lists; they will drive the Move Plan and schedule. In a data center relocation effort, the business impacts of a freeze date can be huge, especially for organizations with high up-time requirements. For example, the big Wall Street brokerages and regional power utilities don’t really go down – ever. In those environments, a “freeze” is regarded as something sacred. Any 24x7x365 operation: the phone company, utilities, energy, hospitals, and certain government agencies each have freeze dates and open window periods when moves can happen. But, they are rarely apparent without some research. The availability of open windows suitable for a data center move will also depend on a host of other key factors. You must strive to identify them all. It's also at this point that we begin to rough-in some contingency dates. What happens if the new facility you are building or renovating isn’t completed on time? With most data center relocations, you can almost guarantee that the new computer room will not be completed on the date the contractor originally gave you. Be sure to develop those contingency dates and carefully plan around a set of realistic availability windows.

Move Day Planning Once all the planning and preparation work is done; after you’ve run through several dry-runs for each major move group, then it is time to make the actual MOVE. Getting everything ready for that special time is called “move day planning.” Move Days will normally be a multi-step process, especially if a new “house” is being built. Prior to actually moving in, it may be decided that rooms need to be painted, carpets replaced, and bathrooms upgraded with new fixtures. A key task will be to clean everything. This is called site prep. In any data center relocation, there is always site prep - if for no other reason than you need the proper electrical and network connections in place before you move in the equipment. Before the data center is ready for move-in, you will have completed a series of final inspections. Any problems noted will have been resolved. And you do have to clean the under-floor areas particularly well. The Certificate of Occupancy will (hopefully) have been granted, and your communication services, auxiliary power, and other details are all done, approved, checked off, and ready to go. For a major DCR project, each Move Day is usually complex enough that it will require its own detailed plan. Next, come the walkthroughs. The equipment movers won't do the job until they can see for themselves what equipment they will need, where everything goes. They will want to know what barriers are present, whether the freight elevators are as described, and most importantly – is there somewhere to take a break, plug in a radio, and listen to the game while they work into the night.


37/71

As you do the pre-planning for a data center relocation project, walkthroughs are critical, since you will have a lot of people to coordinate. Finally, when all of this is done, the move happens. In the overall scheme of things, there is relatively less activity for you during the actual move period, compared to all the discovery, analysis, and planning work that makes the move days possible. You focus here is on making sure that the schedule is adhered to, and that exceptions are handled immediately. In relocation terminology, we call the actual moves “cutovers,” since we are "cutting over" the workload from one site to another. A successful cutover is the result of teamwork and planning.

A Note on Moving Equipment: The important process of un-racking, packing, transportation, un-packing, and re-installation of expensive computer equipment is best done by I.T. logistics professionals. Your DCR planning SME knows the right companies for this critical job in your region, as well as the importance of having this part of the DCR project go smoothly. You can delegate this critical job to skilled professionals who will insure, package, move, unpack, and pre-position your equipment in the new data center – right where your floor plan says. The vendors and your own I.T. teams can then re-assemble racks and cabling and bring up the systems for a test run. Your Move Group Plan will have everything detailed, including specific instructions on when and how each system is re-racked and fired up, and then brought back online.

You could load it all into the back of a U-Haul truck (or, somebody’s pick-up), toss in a few manuals you think you might need, and drive everything over bumpy roads on some dark and stormy night. Stop laughing! Really. When the pressure to cut costs and “get it done…” reaches a certain frantic level, something close to the latter scenario becomes all too real, especially in smaller, departmental, or “office” projects. We have seen it happen. Of course, if the decision was made to move one room a night, and in a staffer’s pickup truck, there would be a lot more flexibility as to what nights you might have to choose from. Just check first to see if Freeze Dates interfere and if The Risk Plan that you created also covers traffic tickets, sleep-deprivation-induced accidents, and other assorted road hazards.

Power, Heat & Fire Suppression Power consumption will increase, even if the newer systems are more energy-efficient. The temptation will be to pack more 1U and 2U units into a new rack, which increases heat, which increases HVAC requirements, etc. Heat is the primary danger in a data center and these machines produce immense amounts of it. One of your many tasks will be to validate that the facilities designers got it right. You must know that there is sufficient cooling capacity and power for current and projected growth. You don’t want to have someone a few years from now ask why the cooling or power generation infrastructure isn’t adequate to support the newest high-density expansions in systems and storage.

Or…,


38/71

Budget Games Modern fire suppression systems design, implementation and maintenance for the data center is as much an art as science. This task is best left to certified vendors and installers. In a raised floor environment, the above-floor space will be well provided for. Check to be sure that the under-floor space is also covered in the design. Today’s data centers don’t mix all that well with water-based (sprinkler) systems, but surprisingly there are still quite a few of these installed, even in fairly new buildings. If your data center is being renovated, you may need to check to be sure that the walls go all the way up to the ceiling. Don’t be surprised if they don’t, especially where false or suspended ceilings are installed. Gas systems require a sealed room. You will also need to have an environmental engineering report. Are there external fuel tanks for the emergency generators? Are they in compliance with the new generation of EPA standards?

Financial planning in the data center is not for the faint-of-heart. Typically, as much as 85% of the total enterprise I.T. budget is spent there. That means the remaining 15% is for the thousands of desktops, laptops, Blackberries, local printers, and other peripherals. Millions, perhaps hundreds of millions of dollars, euros, yen, or pounds are committed yearly in an unending race to support business growth. Because the data center is the major cost center for I.T., your DCR project will receive weekly scrutiny at the highest levels of the enterprise. You will be required to develop and present concise summary briefings. These will cover the milestones and current project status. But what everyone will really want to know is how the budget is going. Therefore, one of the first things you will need is the support of your financial analysis and controls group. They have the skills and presence at the C-Level to develop the right financial message. When you are involved in the pre-planning for a relocation or consolidation, try to focus on the expected benefit. That will be detailed in the business case that started this whole effort. You may discover that the project’s ROI is somewhat (or even wildly) optimistic. That’s not necessarily a bad thing because this ROI process is in many ways still pretty much a SWAG exercise at this point. ROI projections can be used to justify expenditures. They can also be used to justify DCR budget cuts. And that, naturally, brings us to the most difficult aspect of your project – doing DCR with budget reductions.


39/71

Doing DCR With Less Having worked in the corporate or government environment for some years, we all know that come September through December (for business), and June through October (for many government agencies), the annual budget cutting exercise gets underway. That’s just the way things are. Knowing this, you can begin to develop risk plans to cope with the inevitable. The actual migration may need to be extended into the next fiscal year. If so, weather now becomes an issue. Also, the year-end freeze dates and holidays may reduce the available days for migrations. However, the key difference between a DCR and most other I.T. projects is that too often the move dates cannot be changed. The lease may be up on your existing facility, the company might have sold the building. Or, a merger & acquisition (M&A) has mandated consolidation of your data center(s) with the new parent company’s facilities in one or more locations. So, if budget cuts require schedule or scope changes, the project must restart its analysis and planning phases (at least in the affected areas) to cope with the new state of affairs.

Cultural, language, political, economic, social, and technological implementation differences can add frustration to the project. Or, they can enrich your experience in unimaginable and delightful ways. We have planned more than a few of these, and the challenges are significantly more complex than the usual DCR project. But, also more fun when planned well.

DCR Checkpoint – Why Move? Data centers are relocated or consolidated for business or technology reasons. A business-driven DCR can be the result of changes in the business – growth due to new markets, products, and services. It can be caused by mergers & acquisitions or divestitures; or perhaps an increase in data retention requirements due to changed regulatory environments. Technology-driven DCR is usually the result of the continual evolution of I.T. infrastructures. Increased server density, higher energy costs, changes in architecture due to facility expansion, and the need for high-availability systems each can require relocation to new facilities or consolidation of systems in-place. Whatever the reason, migrating, or consolidating a data center every few years seems to be almost an inevitability, given the rate of change in the business sector.

International DCR Projects For those who relish high adventure and travel to exotic places, there is good news. Data centers in large multi-national companies are migrating, relocating, and consolidating as never before. Many of them are moving offshore. Of course, to get the scope and complexity index right, you will probably need to multiply the already formidable challenges found in a domestic DCR by a factor of ten.


40/71

About Consolidations We have focused on the Relocation part of DCR. But far more “consolidation” projects occur than relocations. So, what are the key differences? A consolidation is normally undertaken to reduce costs. This means that financial analysis is critically important in the definition, planning, and implementation of a Data Center Consolidation (DCC). A consolidation is easier to plan for than a major relocation. But, it’s still not a trivial exercise at all. Relocations involve a wider range of systems and services, but major consolidations can be every bit as challenging.

The applications to be impacted include CRM, HR, and Financial – all of them critical. This means that the consolidation must occur in carefully managed stages, with ample opportunities for testing and systems assurance.

For example, a company is consolidating approximately 5,000 Wintel servers as it migrates from the Microsoft Windows 2000 platform to Windows XP. Windows Vista is not being considered because it is still too “new.”

It also means that data security (physical and logical) is going to be extremely important. Plan for that condition, too.

The DCR project will impact desktop systems to the extent that they require consolidation of email and productivity application servers.

Virtualization tools will be utilized to re-host the applications, once the new servers are installed. Because multiple applications will be running concurrently on the same system, careful attention should be paid to DR/COB

A new Citrix thin-client environment must be supported, requiring the migration and consolidation of several servers. There are also about 250 systems that are still running Windows NT 4.x that must be retired.

This is the time to look at the new generation of multi-core servers. Consideration of memory and hard disk needs, as well as LAN requirements is also warranted.

The enterprise-class UNIX systems are out-ofscope, as are the mainframes and midrange (AS/400) systems.

Generally, you will begin with an overview of the key applications and systems targeted for consolidation. Possible candidate may include those acquired through M&As.

The target goal is to cut the number of Wintel servers by around 30%. That may not seem to be an aggressive goal to the inexperienced, but there generally isn’t going to be a 2-for-1 reduction in the application hosting footprint. Servers, applications, and virtualization tools aren’t that efficient yet.

Once the high level list of potential targets has been agreed upon, a step-by-step iterative discovery and detailed analysis process begins. What you will need to determine are the specifics of each system and application, its current and projected growth trend, technical requirements, and other details of procedures and system configurations.


41/71

Determination of the host “footprint” – the amount of computing resource necessary to run the application will be next. This is important because consolidations increasingly mean hosting multiple applications of a single server. You need to determine that the target system will have adequate capacity. You must also be sure that the target facility can actually support the added systems: physical space, cableways, power, cooling, etc. Consolidations will also impact the network. Faster systems with gigabit LAN cards may add to the load. Planning and capacity utilization analysis is required here. Faster servers (racks or blades) also produce heat. Cooling capacity must be examined. The newer racks have option in-rack cooling options, which are nice. But, remember, the excess heat still has to go somewhere. If that somewhere is down into the under-floor plenum, can the main cooling systems handle the added burden? An aside: Has anyone actually looked at the under-floor space recently? It’s probably pretty dusty – and crammed full of fossilized cabling left over from the last relocation or consolidation...

When you move, use all new cabling. Heat can damage and degrade the performance of any cabling over time. This means that someone will probably need to be sure to “measure” the amount of new cabling that will be required. Cabling experts will help with this. Also, when you finish the job, the abandoned cabling will probably have to be removed from the old under-floor and overhead spaces. Most local fire codes require this as a safety best practice.

Consolidation Checklist Here are a few of the many things that you may want to consider when planning for a server consolidation (mainframes are somewhat more complex and require vendor support): • • • • • • • • •

Capacity Requirements (current) Capacity Requirements (new) Data Requirements System Configuration Requirements Target Applications HVAC & Power Requirements Facilities Impacts (is there room to add the new equipment before the old is decommissioned?) Technical Services Other

While it doesn’t occur nearly enough, periodic (more than once-a-decade) cleaning of this space is essential to reduce fire hazards and improve cooling. When you are down under the raised floor, take a look at the cabling. There are miles of it. And, untangling all that you see down there (and the miles and miles that you don’t yet see) would take just short of forever.


42/71

Server Consolidation & Virtualization A detailed coverage of this important topic is beyond the scope of this Guide, but it is a valuable tool for consolidation of server farms and applications, particularly on the Wintel and UNIX/Linux platforms. Server consolidation is all about cost reduction. Because application vendors often license by the box, as well as by the number of concurrent users, any reduction in the number of machines running an application can result in serious cost savings over time. VMWare is the leading provider of the platform solutions for a number of architectures, including UNIX, Linux, Windows, and others. There are also several other products available – this is a rapidly growing market. Microsoft offers its own virtualization solution – Virtual PC. While not as sophisticated as the VMWare solution (and only for its own platforms), it does have one advantage – it’s free (for now). Virtualization, as a technology, has been around for a long time. First, on IBM mainframes, and later on various mid-range platforms, its usage (and usefulness) has exploded with the rise of cheap, powerful Intelbased servers. Led by Google, which has pioneered the use of massive (generic) server farms, modern consolidation approaches rely heavily on virtualization and sophisticated architectures to enable the use of lots of “cheap iron” in the enterprise.

Special Considerations Now that you have some idea of the relocation process, let’s look at some areas that require special attention.

Budget As you were reading about relocation, you may have noticed that while the process is linear enough, the level of detail is revealed in a “topdown” manner. This makes for some interesting business challenges. The first and foremost challenge is budget — which is one of the very first things that you are going to be asked to come up with. Unfortunately, you will not have all the information you need to even start on a decent budget until you have completed The “How” phase. Even at this point, it is recommended that you set aside a larger than normal contingency fund — at least twenty-five percent. This is due to the fact that the technical detail isn’t really developed until the “detailed Planning phase (when you start building the site preparation plan and cutover schedules). Invariably, the more detail you obtain, the greater the chance is that additional costs will be uncovered. This is neither bad nor good; it is just part of the normal process. It’s like building a house — even if the number of interior walls is known, you won’t know exactly what it costs to build them until the quantity of studs, nails, sheet rock, tape, outlets, etc., is determined.

This innovative approach, when planned and implemented well, can result in better ROI, improved performance, faster and safer migrations, and more reliable DR/COB environments.


43/71

Communications Another issue with developing project details in a top-down manner is that something may be uncovered that significantly changes or invalidates your original relocation strategies. This can range from a minor inconvenience to a major rework of the entire project. The best way to minimize the impact of such an event is to catch it early. The best way to uncover problems early is through good communications. The better the staff knows and understands the project, the fewer problems you will experience. One good way of doing this is through an online discussion forum. This is relatively cheap and easy to develop utilizing existing tools such as an internal web site. But, the forum must be moderated to control rumors and misinformation.

New Construction If your relocation project involves the construction or refurbishment of a new computer room or building, you have a whole new set of problems to deal with. First and foremost are schedule considerations. As mentioned earlier, construction projects rarely finish on time, so you will need to develop extremely good contingency plans for date changes. Additionally, there is a strong likelihood that the facilities department is managing the new construction. While they will have good, strong knowledge of infrastructure, they are unlikely to have a full understanding of the operational issues in an IT department.

And, while general contractors know how to construct or renovate a building, you are almost guaranteed that they have an incomplete I.T. perspective. This can lead to a situation where the new site’s design or refurbishment is less than optimal for your needs. A simple example is computer room layout. An I.T. department needs an open and easily managed space that maximizes equipment density. This means that air conditioners and power distribution units need to fit into the equipment layout, not the other way around. There also needs to be adequate office space, and room for expansion. You may not have input at this point in time, but be aware that computer rooms can be constructed with either (or a mix) of floor types: raised or solid. Each has its adherents and detractors, but both have advantages. Raised floors have a vast and hidden plenum for cooing and cabling purposes. They can also be a potential fire hazard because some fire suppression systems may not adequately cover this area. Solid floors, on the other hand, must have cabling exposed, running from the ceiling. A key advantage here is that you can expect to support much higher floor loads – heavier equipment can be moved around without worrying if you will punch through the raised floor and damage sensitive and very expensive systems. For most I.T. organizations, it a given as to which environment they will live in. Left too much on their own, facilities planners and the general contractor just might opt for a layout that best fits their plumbing, wiring, or fire suppression needs; not necessarily your requirements. It is critical that the IT department has someone designated to interface with facilities and the general contractor. That person’s mission is to ensure that you end up with the best data center possible, not just a standard building “refurbished” to hold a bunch of computers.


44/71

Staff Relocation If your project involves relocating more than twenty-five people, then you are going to be faced with some significant HR, management issues, and the usual technical problems. As the number of people to be relocated grows, the problems associated with them grow at a disproportionate rate. Generally, corporations deal with this issue by clearly dictating the terms and conditions of the move. While this sometimes works, it has the negative effect of reducing moral and alienating the resources that your company’s business depends on. It is much more effective to treat the staff relocation on the same level as the equipment relocation. This includes equal time from the project manager, good communications, and a high level of involvement in the project.

There are considerable technical issues in doing this, and the larger the existing network, the more difficult this is.

At a minimum, each department should have one person assigned to the staff relocation team. Not only does this give them a voice in the project, but it also gives them a sense of ownership — success or failure depends on them.

Also, there is very little opportunity to relocate existing equipment (because the existing network is still in use), so there is usually a considerable dollar investment in new hardware.

Networks

Finally, procurement of network components and circuits can require long lead-time items. This means that the new network planning needs to happen as soon as possible and will most likely involve considerable input from your network vendors.

Nowadays, most businesses maintain a high level of online availability with their customers. This immediately presents network concerns. You will need to maintain access to your host systems regardless of where they physically reside. A relatively new factor in network design is the network latency created by the distances that separate today’s data centers. That can impact application performance and even design. Your DR/COB plan will need to address this in terms of the RPO and RTO requirement (See the DR/COB section). The norm in most data center relocations is to create a new network backbone at your new location and then connect it to the existing network.

Project Staffing Levels A good IT organization will have technicians that are willing and able to help in the design and build out of a new data center. Unfortunately, every one of these people will have an existing job, with very little spare time. While it may be expected that they make a special effort to support the project, there is a practical limit to how long this special effort will last. Relocations are time and resource intensive. It may take well over a year to plan and execute the relocation of a large IT operation.


45/71

The TenStep DCR Plan DCR Checkpoint – Who Is Involved? Because a data center relocation or major server consolidation is so complex, you need to be sure that the right teams are in position to help you succeed. But who are they?

Once the project is defined and the core team assembled, there are certain activities that are essential. Here is a simple list of the ten most critical tasks that any DCR project manager must take care of: 1.

The I.T organization will drive the project; it is the group with the technical and operational resources. Business units, major application and services users, and even key customers will also need to be represented at critical points. Your vendors and services providers must be involved because they have the expertise needed to de-install, re-install, and re-certify systems under maintenance warranties. Also, since your data center will have equipment from many vendors, their experience in data center migrations will be added assurance of success. Data center relocation and consolidation specialists are essential because a DCR is an infrequent event for most I.T. organizations. Except for the largest and most dynamic companies, those undergoing constant major re-organization or M&A activities, very few I.T. groups have the in-house expertise, best practices, tools, and analysis capabilities of a DCR specialist firm.

You won’t get the detail required without considerable effort. So, if this comes easily, you haven’t done it right. 2.

Security - the dominating characteristic of a large DCR is activity. Everything is in motion at one point or another. A good security plan will help ensure that data doesn’t disappear, that unauthorized people are kept away, and that the business remains un-compromised.

3.

Planning – from the early +pre-planning” phase through detailed DCR planning, to schedule development and the move days, you will spend far more time and effort on this area than anywhere else.

4.

Budget – A DCR is expensive, very expensive if what you are relocating or consolidating is a major facility. The DCR budget must adequately cover new construction, renovation, site closure, equipment, staff, tools, and outside expertise from vendors and DCR specialists.

The relocation specialist team will also know how to manage the details, from project startup through planning and to the actual move. The DCR SME’s will be able to coordinate the work of many third parties, including: • • • • • •

Vendors Insurance Security Transport Equipment de-certification & re-certification DR/COB

And many other aspects of the project that will enable the DCR project manager to achieve his or her objectives.

Inventories – hardware, applications, and every component that will be relocated or consolidated. The inventories need to be detailed, accurate, and complete – with equipment model and serial numbers, configurations, replacement value, vendor contacts, and a Visio diagram of how each system is de-installed (before) and reinstalled (after).

Managing the budget and keeping your executive management well informed are major challenges. 5.

RFPs, SOWs & Contracts – Vague RFPs make for poor SOWs. Poor SOWs make for terrible contracts. Take the time to work with DCR specialists right from the start to develop the right RFPs and Statements of Work (SOWs) for your project.


46/71

6.

Use the DCR Specialists – Selection of the right type of DCR specialist for each critical area is important. But, do you need one single company to do it all? That depends on you. If your internal DCR teams lack specialist skills, whether in planning, schedule (Move Domain & Move Days) development, or equipment deinstallation, moving, and re-installation, then you will need to acquire the services of a number of specialists. Whether your primary DCR vendor states it or not, all of them use partners because nobody can be the best at every aspect of such a complex process. Select the best planners first; they can help you choose the ideal movers and other professional services required.

7.

Plan the Move, Move On Plan – Moving equipment is a critical part of the project. Systems must be broken down, packed, transported, re-assembled, tested, and recertified by the vendors. Racks and other equipment support systems must be ready, utilities and communication services need to be ready, and people have to be migrated from where they are now to where they will be.

8.

Prepare the New Facility, Close the Old One Inspections of any new or renovated data center must focus on more than the technology being installed.

9.

Backup The Data – Your backup media will work (it ALWAYS does, you NEVER have problems), but just in case it doesn’t this time, you need to have a recovery plan. And that plan must be thoroughly tested, with practice runs conducted at regular intervals during the detailed planning phase. Virtualization tools can be utilized here to host systems remotely. But even there, a DCR-based recovery plan is essential. And it’s not the master DR/COB plan that everyone has, but too few organizations test enough to assure that it will function as expected when required.

10. Migrate – when it’s time to move, stop planning and move. This is the moment when careful planning and capable project management result in flawless Move Days. When everything has been relocated or consolidated, the old systems retired, and the vendors have re-certified their products, you can take that well-earned vacation day. And then get ready for the next big project because Management now has a better understanding of the business value of great project management.

You must ensure that fire suppression systems are ready, tested and approved. Cooling systems must be adequate for projected growth. Utilities must be in place and operational. And, the place must be clean – very clean. While you are necessarily focused on the destination, you must also do what is necessary to de-commission equipment that won’t move and close the old data center facility.


47/71

A key part of any successful DCR is the updated, tested, and thoroughly practiced DR/COB Plan. Why? Because, should anything go wrong in your migration, the DR/COB plan developed for the DCR will tell you what to do to get back on schedule.

DCR & Disaster Recovery If you are new to recovery planning, make sure that you research the subject thoroughly before embarking on a disaster recovery project for your DCR project.

DCR PrePlanning & Disaster Recovery / Continuation of Business (DR/COB) Disclaimer The following project outline is provided solely as a guide. It is only intended to be "one example" of requirements for a disaster recovery project plan. It is not, by any stretch of the imagination, the only way to set up a DR/COB project plan for DCRs. A DCR project has been likened to a “controlled’ disaster recovery. While an exaggeration in many ways, the analogy demonstrates that data center relocations (and major consolidations) must never be planned in a vacuum, or with less than the best talent available. This section of the Guide presents this subject at a high level only. While the subject of recovery is critical to any successful DCR project, it is best done by subject matter experts.

Consider engaging a consultant (internal or external to your organization) to help you in your project planning effort. Like the DCR, disaster recovery planning is not a two-month project, neither is it a project that once completed, you can forget about. The key point we want to make in this section is that the master DR/COB plan for your organization is not necessarily going to be the right solution for the DCR project. You should consider adapting it to the specific needs of your DCR project. A key difference between a “master” DR/COB plan and those activities specific to a DCR recovery plan is the concept of “availability.” We’ll provide an overview of that area later in this section. There are, naturally, many overlaps. But the goals are different enough that the master plan just might not be all that helpful if things go bad. To appreciate these important differences, we need a quick overview of the basic components of a DR/COB plan. Then, we’ll discuss how you adapt it to DCR projects through a focus on “availability” planning.


48/71

DR/COB for DCRs

DCR Business Resumption Plan

Disaster Recovery solutions can be presented in the form of a Disaster Recovery Architecture Blueprint (DRAB). The DRAB is the over-arching IT document that includes the DR Plan, the DR Solution, the DR Operations Process, and the DR Test Plans (see table below).

The primary objective of a DR/COB Plan for the enterprise is to enable your organization to recover faster and to resume normal business operations when something goes terribly wrong during the normal course of business operations. •

The DR/COB plan is your insurance policy, so keep up the premiums. An effective recovery plan is a living recovery plan and must be maintained and tested/exercised regularly.

• •

Program Description Pre-Planning Activities Vulnerability Assessment DCR Requirements Business Impact Analysis Detailed Requirements Definition DCR (DR/COB) Plan Development Testing/Simulation Program Maintenance Program Initial Plan Testing and Plan Implementation Planning Scope & Objectives Project Organization and Staffing Project Control Schedule of Deliverables Resource Requirements

Adapting the “master” DR/COB process and plan to fit a DCR role is one of the more important tasks that you must do.

•

o

o

Regardless of the cause, when the DR/COB plan is triggered, fast deployment and quick recovery is essential. The organization must assure that critical operations can resume normal processing within a reasonable time frame. However, the primary objective of a DCR DR/COB plan is to enable fast restoration of critical services after a move. This may seem to be an arbitrary difference, but it is important. NOTE: Check and re-check your data backup procedures and media. Nothing is more disruptive that needing to restore tapes, only to discover that they don’t work as expected.

This is a “standard” outline for an enterprise DR/COB plan. It must be refocused, with a narrowed scope, for use in developing a plan suitable for DCR projects.

DRAB

o

Triggers for implementing the plan can be natural disasters such as floods and storms. Social disasters can include terrorism, civil unrest, and disease epidemics. Technical disasters can be the result of infrastructure failures such as power or communications outages.

Disaster Recovery Plan – Document that outlines (at a high-level) the process of implementing the DR solution in accordance with preestablished DR requirements.

DR Plan

DR Solution

Disaster Recovery Solution – The disaster recovery method(s)/process/tools presented in the DR Plan in meeting customer’s DR requirements; for example, using OVSM to replicate data.

Implementation Plan

Implementation Plan – A component of the DR Plan, it outlines the Stepby-step process of building the DR Solution.

DR Operations Plan

DR Test Plans

Disaster Recovery Operations – Document that describes the step-bystep DR failover process in the event of a disaster. The document also presents the fallback (recovery back to production) process as well. Disaster Recovery Test Plans - Document that describes the process and procedure in testing the DR Plan. It addresses test goals, strategies, and acceptance criteria.


49/71

The primary objective of the DRAB blueprint is to provide (in a cost-effective and timely manner) continued services to the Business in the event of a disaster. Therefore, the goals of the DRAB are:

Identify weaknesses and implement a disaster solution Minimize the duration of a serious disruption to business operations Facilitate effective co-ordination of recovery tasks Reduce the complexity of the recovery effort

It is important to note that the DRAB is a living document. Both the information processing and the business environments are constantly changing and becoming more integrated and complex.

DR/COB Methodology for DCR Leverage the organization’s master DR/COB methodology, but refocus it on narrower requirements. When implementing your methodology, consider the following key points:

Recovery plans must keep pace with these changes. Continuous testing/exercising of plans is essential if the business wants to ensure that recovery capability is maintained in such an environment.

The business also must ensure that staff members with recovery responsibilities are prepared to execute the plans.

Who Should Do The DCR DR/COB Plan?

Be sure that Management understands the effort required to develop and maintain an effective recovery plan during and after the DCR migration event(s) Define recovery requirements from the perspective of restoring business functions quickly Document the impacts of an extended downtime on operations and key business functions Focus on risk identification and mitigation to minimize the above impacts Build your DCR DR/COB project teams so that a balance between I.T. and business participation is achieved Produce a DCR DR/COB plan that is understandable, easy to implement and maintain Plan for a straightforward transition from the DCR DR/COB plan to the master plan once the project is completed

The I.T security group is usually assigned the responsibility for providing disaster and contingency planning. But for a DCR, this may result in recovery plans that are not fully responsive to the needs of the business during a relocation project. Within the context of the DCR, recovery planning is foremost a business issue. Only secondarily is it an I.T. problem. The development of an effective DCR recovery strategy must be a cooperative effort between I.T. and business. Key members of the planning team should include operations, communications, as well as users of these systems.


50/71

Such a plan requires the cooperation of management from all areas of I.T., as well as every key business area that is supported by I.T. Since recovery planning is a very complex and labor intensive process, it requires the commitment of technical and business staff, as well as appropriate funding. Incorporation of DR/COB planning for your DCR project should begin in the pre-planning phase. It is here where you define key objectives, identify the important risks, and sketch out recovery strategies. A detailed coverage of an appropriate DR/COB methodology for DCR is beyond the scope of this Guide, but at a very high level, it might look like this: •

• • • • • •

Project Initiation o Scope o Budget o Staffing o Risk Assessment o Security Assessment Requirements Definition Business Impact Analysis (BIA) DCR DR/COB Plan Development Test/Simulation Implementation Continuity/Maintenance

In this Guide, we will cover these only at a high level for informational purposes only. DR/COB specialists should be consulted for more detailed insight into your environment. The Requirements Definition phase is where the DCR plan will begin to differ from the “master” DR/COB plan. In addition to standard PMI project definition and management practices, the proposed DCR project plan will consist of the following, not necessarily mutually exclusive, phases described below:

Phase 1: Detailed Requirements Definition During this phase, a profile of recovery requirements (including two key driving elements: (RTO & RPO) and a risk analysis will be developed. This profile will be used as a basis for building and implementing the recovery and must be completed prior to starting the DRAB. Some key elements of the profile that will be considered and defined are:

Recovery Time Objective (RTO) A business impact analysis, as well as the calculated cost of downtime, provides insights into the Recovery Time Objective (RTO), an important statistic in business continuity planning. It is defined as the maximum amount of time that an IT-based business process can be down before the organization starts suffering significant material losses. RTO indicates the downtime tolerance of a business process or an organization in general. This tolerance is proportional to the missioncritical nature of the business, e.g. a stock exchange system would have a RTO of nearly or equal to zero.


51/71

Guiding Principles for Recovery Point Objectives •

Recovery Point Objectives are defined as the maximum amount of data loss (expressed in time) a system, application, or business process can tolerate following a catastrophic failure or outage.

•

Recovery Point Objectives are established based on the certainty that data loss will occur in a catastrophic event or outage. Data loss is defined as any data/transaction that was processed but unrecoverable following an unexpected interruption.

•

Recovery Point Objectives are assigned by the business owner. Establishing recovery point objectives requires collaboration with supporting technology, business continuity, and LOB process groups.

•

Recovery Point Objectives can impact the Recovery Time Objectives depending on an individual application’s requirements to recreate lost data before making the application available to the customers.

•

Lines of business need to ensure their technology partners have documented a way to identify the amount of data loss after a catastrophic failure or outage.

•

Lines of business need to document the recovery steps to recreate the lost data or acknowledge the data will not be recovered.

•

Existing processes will be utilized to assure consistency and provide oversight in the assignment and maintenance of the recovery objectives.

Recovery Point Objective (RPO) Recovery Point Objective (RPO) is another important statistic for business continuity planning and is calculated through an effective business impact analysis. It is defined as the maximum amount of data an IT-based business process may lose before causing detrimental harm to the organization. RPO indicates the data-loss tolerance of a business process or an organization in general. This data loss is often measured in terms of time, for example, 5 hours or 2 days worth of data loss. To use the stock exchange as an example where millions of dollars worth of transactions occur every minute - one can expect that an RPO of zero would be ideal (note that RPO does not include the loss of data in transit), and that, with proper design, this objective is possible if the target recovery site is within 50 miles of the production data center. Unfortunately, in the case of financial institutions, based on the events of 9/11, the Office of the Comptroller of the Currency and the Federal Reserve Board modified their position around proximity risk based on geographic location. In other words, Disaster Recovery data center sites and Productions data centers distances must be greater than 200 miles. Because present replication technologies cannot reliably accommodate an RPO of zero at those distances – there will be some inevitable data loss that must be planned for.


52/71

Risk Analysis (RA)

Other Profile Elements

RA identifies the risks that a DRAB must minimize; there are three elements to consider: threats, assets, and mitigating factors.

These will include, but are not limited to, HW/SW, data (financials, personal, etc…), security (physical and logical), external feeds, application bindings, facilities, personnel, etc…

Threats/Disasters are events or situations (scenarios) that would cause financial or operational impact to the organization. These are measured in probabilities, such as "may occur one time in 10 years.” Each threat has a projected duration of time that the business or operation would not be able to function in its normal manner, if at all. Typically there are two types of disasters – 1) Natural disasters such as storms, i.e. Hurricanes, earthquakes, etc… and 2) manmade ones such as computer viruses, human error (mistakes in system administration), terrorism, etc.

Phase 2 Plan Development & Maintenance Program During this phase, recovery solution components will be defined and documented. This phase also includes: • •

Both of these types of disasters manifest as physical and data access disasters. Assets are composed of the physical assets that are owned by the organization and its financial assets as well. Revenues lost for the duration of the incident, additional costs to recover, fines and penalties incurred, lost good will or competitive advantages are each necessary components in the assets figure. Mitigating Factors are the protection devices, safeguards, and procedures in place that reduce the effects of the threats. They do not reduce the threat; they only reduce the effect of the threat.

• • •

Identification and implementation of changes to user procedures Upgrading of existing data processing operating procedures (e.g. backup procedures, patching, etc…) required to support the selected recovery strategy Vendor contract negotiations (with suppliers of recovery services, i.e. Telco, DNS services, etc…) Definition of Recovery Teams, their roles and responsibilities Initial Disaster Recovery Operations will also be developed during this phase

Maintenance of the plans is critical to the success of an actual recovery. The plans must reflect changes to the environments that are supported by the plans. It is critical that existing change management processes are revised to take recovery plan maintenance into account.

Examples of mitigating factors in use include: • • • • •

Uninterruptible Power Supplies (UPS) Generator backups for replacement power Advanced fire suppression systems to control the spread of fire Access card readers to control physical access to company space. Remote recovery sites


53/71

Phase 3 Proof of Concept, Research, and Implementation Proof of concept testing and implementation strategies will be developed during this phase. Once plans are developed, tests of the plans will be conducted and any necessary modifications to the DRAB made based on an analysis of the test results. Thereafter the plans will be implemented.

Phase 4 Testing/Exercising Program The planned Testing/Exercising Program will be developed during this phase. Testing/exercising goals will be established and alternative testing strategies evaluated. Testing strategies tailored to the computing and business environment will be selected, and an on-going testing program will be established. As the recovery operations strategies are defined, specific testing procedures will be developed to ensure that the written plans are comprehensive and accurate.

Phase 5 DR Test Plan Execution In this phase, the blueprint will be tested. The approach taken to test the blueprint depends, in large part, on the recovery strategy and availability of resources for both infrastructure and personnel. In addition, testing will be adapted to the target environment to minimize disruption to production environment.

Adapting DR/COB to DCR It is at this point in the adaptation of your organization’s DR/COB plans for the DCR project that you should engage your organization’s DR/COB, Information Security, Legal, and other teams. You should also have (by now) identified and engaged the equipment, application, and services vendors - and technical SMEs that will be necessary to ensure project success. And, naturally, the business units, end-user management, and key customers that will be directly impacted by the relocation or consolidation should already be wellrepresented on the DCR “pre-planning” teams. And, if these are not, why aren’t they? Any master DR/COB plan must be adapted to the specific requirements of a DCR. A key area of difference between the two plans is that while the enterprise DR/COB plan is intended to last forever, the DCR DR/COB plan ends when the DCR project ends. To better understand how and why this DCR adaptation happens, let’s address two key points: Availability, and Recoverability.


54/71

Understanding Availability and Recoverability Availability Within a Data Center - The ability of a component or service to perform its required function at a stated instant or over a stated period of time. Routine – These are business/technical interruptions that are part of every-day processing and are mitigated and returned to normal delivery. Examples include failed component, operator error, and failed change implementation. Non-Routine – These are events or a chain of events that interrupt, cause slowdown, or completely halt processing or service delivery unexpectedly. Examples include CPU slowdown, denial of service attacks, and viruses.

Characteristics • • • •

Routine or non-routine events Technology can be employed to reduce the potential data loss to almost zero Availability objectives and capabilities are addressed in systems design and are used by technology partners Availability objectives are demonstrated by achieving and maintaining service level agreement

Recoverability Between Data Centers, i.e. DR - The process of restoring business services after a disruption of service by transferring Information Technology (IT) resources to secondary or backup locations. Catastrophic/Extraordinary – These are the effects of a natural, technical, or human disaster that would warrant the enterprise, or impacted part of the enterprise, to operate in a disaster recovery mode. Examples include natural disasters and data center failures.

Characteristics • • •

•

Catastrophic/extraordinary events Technology can be employed to reduce the data loss to a minimal amount Recovery objectives and capabilities are addressed in systems design and are used by technology and business continuity partners Recovery time objectives and capabilities are validated through annual testing


55/71

The following two Key Principles are important concepts to be understood before determining availability levels: •

Availability is different and completely unrelated to Disaster Recovery / RTO determinations

The business lens used for what would be reasonable to accept in a “true disaster” has no relevance in normal processing. •

Availability is not purely the Production Service Level Agreement for an individual application

The objective for this activity is to measure the services being provided (which could depend on multiple applications being functional). From a move perspective – we would rely on the availability design of a system to recover from a disaster during a move.

DCR & Data Center Capacity Planning This section will provide a brief overview of capacity planning and its role in a data center relocation or consolidation project.

Measure, Test, Trends Capacity planning for any data center environment is generally defined as the: • • • •

Measurement of present systems loads Tests for anticipated loads Analysis of usage trends over time Determination of projected needs and cost

The objective for DCR capacity analysis and planning is to determine two things: how much usable systems and data storage capacity is available before and after the move; and how fast the infrastructure is growing. These two measurements can then be useful in determining the functional design of your new facility, as well as indicating to Management the velocity of budget changes over time. The goal of this activity is to more accurately project computer and data resources that will be needed at some point in the future.

Tools & Techniques No DCR pre-planning exercise would be valid without some consideration of capacity analysis and planning. However, a detailed discussion of capacity planning for DCR is outside the scope of this Guide. If you are relocating a data center to a new facility, this is an ideal time to determine how fast you are expanding your information processing requirements; whether the planned facility will be large enough to handle current and projected needs; and where capacity bottlenecks are likely to appear in the near term.

The use of a variety of tools and methods can help to automate as many of the tasks as possible and reduce IT workloads. In the mainframe and enterprise server environment, systems are expensive and purchasing cycles must be allocated over several budget cycles. Capacity planning is crucial to the longer term budget planning needs of the organization. In the “generic” server farm environment, capacity planning horizons are as short as three to six months; and often much less especially when management needs to act on new business opportunities, implement regulatory mandates, or close M&A activity.


56/71

As more organizations develop regional data centers, both in response to new Federal and state regulations and to exploit business opportunities, the role of capacity planning for data center relocations and consolidation increases in value. Enterprise systems capacity management and planning has been a staple of Fortune 500 companies for a long time, and they are more likely to commit the budget, staff and tools to get the job done right. For the mid-range data center, the availability and cost for suitable capacity planning solutions and subject matter expertise have become affordable. For the smallest data centers, the process is just as useful; and free tools are available online. Of course, capacity planning still takes a considerable amount of IT staff time. That is why too many I.T. groups simply opt for acquiring additional hardware and software when performance starts to suffer. It takes far less time and the results aren’t that different from the detailed analysis approach, especially for the smaller or departmental data center.

Winging I.T. Capacity Planning What kind of capacity planning is appropriate for a DCR? Do you need to engage in a lengthy and expensive analysis, using the latest solutions, with a similarly large commitment by your I.T. staff? The best way to understand what is really needed for DCR capacity planning is to look at what it will be used for. It’s a high level assessment of current capacity and projected data center capacity growth rates in key areas. Think of it as analogous to building an interstate highway map for driving crosscountry vs. a topographic map useful for backpacking in the wilderness. The level of detail that you need for the highway map is much less than that which is required for the topo map – even though the area covered by the highway map is far larger. For the DCR project, you are building the equivalent of a highway map or schematic that will provide high level capacity requirements for the new facility, with sufficient room to grow for the next several years. That’s all.

DCR Capacity Planning Process The process appropriate to the DCR preplanning phase should be simple and results should be presented at a high level. This is an approximation, not street-level details. Also, keep in mind that it isn’t necessary to perform every task all the time. The process for estimating capacity planning for a data center relocation or consolidation may look like this: Load testing – This should be one of the first tasks that you do, particularly for any new services or for critical systems before you move them.


57/71

You need data to work with to develop a load testing program. Getting the right data can be a challenge, but this is how you find out how a system or application behaves under different loads. Be sure to check periodically to make sure you have the right capacity, or to flag work scheduling, or other load impacts. Monitor Triggering Events – Even in the bestrun environments, rapid change can occur. Sometimes you have to add capacity quickly. DCR pre-planning should include the development of scenarios and risk mitigation plans to cope with doing this efficiently. Consolidation – This activity is useful when you are looking for ways to combine CPU and network workloads. Consolidations here can save money on software licensing and equipment costs. More relevant for a DCR project, capacitydriven consolidations mean that you will have fewer things to move – and a deeper understanding of what’s in the box when you do move I.T.

Your business users may not be able to tell, with the required level of detail, what their usage levels or even peak load periods will be (other than the usual EOM, EOQ “freeze” periods.) Trend analysis of past usage data is helpful, but any recent growth spurts or M&A are likely to reduce the usefulness of projects based on prior data. In conclusion, keep your DCR capacity planning simple and appropriate to the planning phase that you are in. Expect to invest only the time for planning that is in line with the expected outcome. The same advice is also appropriate for other aspects of DCR “pre-planning” – keep it as simple as possible, but not simpler than necessary.

Trend Analysis - It seems rather pointless to do all the work required to move to a larger data center if the new facility will reach its capacity in less time than it will take to earn a return on the investment. Look at the long-term usage trends to develop projections about which systems will exceed capacity or cross the margin of comfort needed to maintain an operational cushion before your absolute capacity limits are approached. Use Modeling Tools – Well developed capacity modeling and load simulation tools can yield more precise information on usage and help define specific configurations for your expected workloads. Use modeling tools for high priority services, but resist the urge to “make a science project” out of it.


58/71

1 23 PMCA’s DCR Project Pre-Planning Templates: Available Now

Pre-Planning Communication Plan Template (Excel) Pre-Planning Project Initiation Presentation (PowerPoint) Pre-Planning Issues Assessment Template (Excel) Pre-Planning Equipment Inventory Assessment Template (Excel) Pre-Planning Application Inventory Assessment Template (Excel) Pre-Planning DR/COB Assessment Template (Excel) Pre-Planning Organizational Matrix Template (Excel) Pre-Planning Task List Template (Excel, Project)

DCR Pre-Planning Guides Several complementary DCR pre-Planning guides are under development. Please check back with us at www.pmca-us.com for information on subjects and availability Each template or guide has instructions on how to use it and what role it plays in the pre-planning cycle. To get the PMCA DCR Pre-Planning Guide templates, simply register at www.pmca-us.com with your name, title, corporate email, and phone number. Or, call us at:

1.954.962.9864


59/71

Planning Your Data Center Relocation Project? Wouldn’t it be nice to have a partner who had expertise based on years of experience relocating data centers very much like yours? PMCA can help. When choosing an outside resource to help, there are a few things you need to check.

How long has this resource been around? Will they provide one point of contact for all the different aspects of the project? Are their partners certified to do the work? Is your resource comfortable in a heterogeneous environment? Will your resource put you first? Will you get the level of experience and technical mastery that your DCR project requires for low-risk, high-value success?

Now that you have a better understanding about the data center relocation and consolidation “pre-planning” role in enterprise I.T., it’s time to go start up your own DCR pre-planning activities. Good luck and be sure to contact us for the Template Kit – and to learn more about how PMCA can help you succeed.

PM CONSULTANTS & ASSOCIATES, INC. Choosing the right DCR project planners and approach is the single most important thing you can do to reduce risk, stay on budget, and assure success.

PM Consultants & Associates, Inc. (PMCA) delivers projects within demanding schedule and budget constraints for Fortune 500 clients.

PMCA also provides Senior Data Center Relocation, Migration, and Consolidation Project Planners and Managers to leading I.T. Consulting firms worldwide.

PMCA offers complete Project Management services, including Consolidation, Migration, and Relocation planning, Disaster Recovery, and Business Continuity planning for your data centers and other I.T. infrastructure projects. PMCA Data Center Relocation Planning services are used by the Fortune 100, government, and leading I.T. consulting firms, because we are globally recognized DCR specialists. PMCA’s DCR and Consolidation specialists have worked in every major industry and for most of the Fortune 100 (and many government agencies) over the past twenty+ years. For more information, contact us at: www.pmca-us.com, or call us at 954.962.9864


60/71

PMCA’s DCR Pre-Planning Template Kit (available now)

DCR Pre-Planning Project Communication Plan This template provides an outline structure for organizing the communication plan for your DCR project. Customize it as required. Provided in the Excel 1997-2003 format.

DCR Pre-Planning Project Initiation Presentation This template provides an outline structure for organizing the DCR pre-Planning activities, and presenting that to the Stakeholders. Customize it as required. Provided in the PowerPoint 1997-2003 format.

DCR Pre-Planning Issues Assessment This template provides an outline structure for organizing the communication plan for your DCR project. Customize it as required. Provided in the Excel 2003 format.

DCR Pre-Planning Inventory Assessment These two templates (Equipment, Applications) provide an outline structure for organizing the Inventory Assessment plan for your DCR project. Customize them as required. Provided in the Excel 1997-2003 format.

DCR Pre-Planning DR/COB Assessment This template provides an outline structure for organizing the DR/COB Assessment for your DCR project. Customize it as required. Provided in the Excel 1997-2003 format.

DCR Pre-Planning Organizational Matrix This template provides an outline structure for documenting the Operational matrix for your data center’s DCR project. Customize it as required. Provided in the Excel 1997-2003 format.

DCR Pre-Planning Task List This template provides an outline structure for documenting the tasks required for a detailed DCR or Consolidation “pre-planning” activity. Customize it as required. Provided in the Excel 1997-2003 and Project 2003 format.


61/71

NOTES:


62/71

PMCA DCR “PrePlanning Checklist Overview

You’re now ready to begin. First, here is a more detailed checklist to help you get better organized for the critical DCR Pre-Planning phase. Pre-Planning means those key activities and dependent tasks necessary before you begin the arduous process of detailed project planning. You’re setting the stage here to ensure later project success, so resist the temptation to skip this phase. Start by putting these tasks into a “pre-planning” project plan (or, order the FREE PMCA DCR Template Kit), but keep in mind that, at this phase, your objective is to determine the high-level scope, estimated budget, and resource requirements for the project. The highly detailed system-by-system and application-by-application analysis and planning comes later: The key areas to focus on in preparation for a data center relocation or major systems and application consolidation include: Project sponsorship & leadership DCR planning & risk assessment Legal, mandates & regulations Project management and execution Budget & financial controls Technology & vendors I.T. management, process, staff Capacity Planning Facilities & infrastructure Security DR/COB plans Each of these areas will require considerable effort and resources to identify, analyze, document, develop appropriate solutions, project manage, and evaluate outcomes. This checklist is a high-level reminder of the key areas that will require your attention throughout the project.


63/71

Detailed DCR PrePlanning Checklist Page 1 Business Case o Business Case Preparation o Executive Sponsorship o Project Leadership o Business Unit Liaisons o Legal & Regulatory Management o Financial Controls & Budget Management o I.T. Operations & Technology Leadership o Vendors & Services Providers o DCR Specialists Communication Plan o Executive Communications Plan o Enterprise Communications Plan o Business Unit Communications Plans o I.T. Operations Communications Plan o DCR Project Communications Plan o External Stakeholder Communications Plan o DR/COB Communications Plan Risk Assessment o Business Risks o Operations Risks o Facilities Risks o Infrastructure Risks o Technology Risks o Resource Risks o External Factors Risks o Project Risks o Budget Risks o Unknown/Orthogonal (at present) Risks Financial Analysis & Management o Project Budget Development & Approval o Financial Controls o Financial Risk Management (project budget cuts & delays) Key Stakeholders o Business Units o I.T. o Customers o Vendors o External, Other Legal, Regulatory Stakeholders o Legal & Regulatory Stakeholders o Mandates o Contract Administration Project Sponsor o Executive o Business Units o I.T. Operations o Project


64/71

Detailed DCR Pre-Planning Checklist Page 2

Project Management Scope o Enterprise Program Management o DCR Project Management o DCR Domain Management Teams Hardware Applications Networks Facilities o Key Domain Inventories Hardware Applications Networks Other o Vendor Contacts Hardware Software Services Facilities Management o Current Site(s) De-Commissioning Renovation & Rehabilitation o New Site(s) Construction Renovation I.T. Operations Management o Data Center Operations DCR Policies & Procedures (including SLAs) Systems Deployment Policies & Procedures Systems Retirement Policies & Procedures Operational Readiness Methodologies Operations & Facilities Processes o Data Center Change Management Consolidation Processes New Installation Processes Migration Processes New Data Center Facilities Commissioning Processes Facilities De-Commissioning Processes o Data Center DR/COB Management Primary Site Recovery Site Regional Sites International Sites Mobile Data Centers


65/71

Detailed DCR Pre-Planning Checklist Page 3 Data Center Relocation Management & Methodologies o Capacity Requirements (current) o Capacity Requirements (new) o Data Requirements o System Configuration Requirements o Target Applications o License Requirements o Vendor Support (virtualized) o OS compatibility Issues o HVAC & Power Requirements o Facilities Impacts (is there room to add the new equipment before the old is decommissioned & removed?) o Technical Services o Virtualization Platforms & Solutions o Applications (new OS or hardware) o LAN o Data Storage o DR/COB Data Center Consolidation Management & Methodologies o Capacity Requirements (current) o Capacity Requirements (new) o Data Requirements o System Configuration Requirements o Target Applications License Requirements Vendor Support (virtualized) OS compatibility Issues o HVAC & Power Requirements o Facilities Impacts (is there room to add the new equipment before the old is decommissioned & removed?) o Technical Services o Virtualization Platforms & Solutions o Applications (new OS or hardware) o LAN o Data Storage o DR/COB Application Management o Enterprise Applications Internally Developed Applications • IBM Host • Server • Network • Externally Hosted • Web COTS (commercial Off-The-Shelf) Applications ERP CRM Database Financial Operations HR Sales Other


66/71


o Departmental Applications Internally Developed Applications • IBM Host • Server • Network • Externally Hosted • Web COTS (commercial Off-The-Shelf) Applications ERP CRM Database Financial Operations HR Sales Other o User / Productivity Applications Internally Developed Applications • IBM Host • Server • Network • Externally Hosted • Web COTS (commercial Off-The-Shelf) Applications Email & IM Database Financial Desktop Productivity (MS Office, etc.) Other Key I.T. Management Domains o Mainframe/Host Systems Environmental Factors Vendors DR/COB Migration Analysis Consolidation Analysis o Servers Systems Environmental Factors Vendors DR/COB Migration Analysis Consolidation Analysis o Attached DASD Systems Environmental Factors Vendors DR/COB


67/71


o SAN/NAS Systems Environmental Factors Vendors DR/COB o Data Storage (Auto Tape, Optical, etc.) Systems Environmental Factors Vendors DR/COB o Printers (Enterprise, Specialized, Departmental, Local) Systems Environmental Factors Vendors DR/COB o Data/LANs Intranets Internet Wireless Access Points Mobile Web o Voice Telephone Utility PABX IP Phones Wireless o Specialized Facilities & Equipment Remittance & Bill Processing Systems Mailrooms Bulk Printing Call Centers Manufacturing Management & Control Systems Shipping & Logistics Systems o Other Systems DR/COB & Security o DR/COB Plan Adapted to DCR Project Requirements o Security Plan Adapted to DCR Requirements o Emergency Services & Hospital


68/71


Utilities o Communications Vendors Telephone (Voice/Data) Multiple-Grid Services Cable, Satellite o Waste Management Services Public Commercial o Water /Sewer Public Commercial o Electrical Utilities Self-Generated Co-Generation Multiple-Grid Services o Natural Gas o Diesel / Propane o Renewable Other (list)


69/71

DCR Project Management Checklist While a detailed coverage of DCR project management practices and methodologies is outside the scope of this Guide, this checklist will help focus your attention on the importance of relying on accepted PM standards. DCR Project Lifecycle Project Integration Management Time Management Cost Management Quality Management Human Resources Management Communication Management Risk Management Procurement Management Vendor Management The size of your DCR project will determine the structure of the PM team. For a large enterprise initiative – or one that will have global activity - you should implement the PMI Program Office model: an overall program manager will coordinate the activities of project managers. Project managers should be assigned to each key domain and/or location (in multi-site DCR’s). Domains might include: Mainframe/Host complexes Server farms DASD and Storage systems Specialized equipment Outsourced systems and services Procurement and Logistics Facilities Voice/Data, etc. The all-too-common practice of assigning one PM to run the entire project may work for small efforts. But, a large DCR will need a dedicated PM team.


70/71

For Information, contact:

PM Consultants & Associates, Inc. 1737 NW 72 nd Way Pembroke Pines, FL 33024

Email: [email protected] Or [email protected] Phone: 1.954.962.9864 www.pmca-us.com

This Guide and the DCR Template Kit were created by senior members of PMCA’s DCR leadership team. Authors: John Sarazen and Pete Bouvier. Contributors: William Cserjes, Burl Minnis, Jerry Seay, Peter Jacklin, and Robert Kuplent.


71/71

PMCA Data Center Relocation (DCR) Pre-Planning Guide

Recommend Documents