Life on the High Wire: How to Move 12,500 Global Users From Notes to Office 365 With Minimal User Disruption

Key Findings

• A dual-hop migration from Notes/Domino to an on-site version of Exchange, and then an upload to Office 365, added complexity, but ultimately proved to be the right approach.
• A key challenge was to collect detailed information on all worldwide device configurations prior to migration, including operating system and version, memory and disk configuration, version of Notes and the site location.
• Flowserve migrated between 1,200 and 2,100 users per week over eight weeks. The company established a global schedule and migration at each site starting at 17:01 hours (local time) to gain maximum migration time and minimize user disruption.
• On the whole, users are quite pleased with the migration and the email component of Office 365 is working well in steady-state mode.

Recommendations

• Pick a migration partner with ample experience with Office 365.
• Clean up the old email system prior to migration and consider a migration to an on-site Exchange 2010 version to ease the final mailbox migration to the cloud.
• Focus significant resources on end-user communication, training and support.
• Understand that Office 365 is relatively new and that there may be some complications due to unknown bugs, varying upload capacity and immature change management procedures.
• Network remediation, such as adding points of presence and prioritizing network traffic, is likely to be needed with a move to cloud-based email services.

Introduction

Wholesale migrations of organizations' collaboration portfolios rarely occur — once in a decade is usual. Reasons for the change of vendor typically involve user satisfaction issues, aging technology and risk abatement strategies. These migrations can be complex and expensive and one of the key goals is to make it easy for users. This document details how one company succeeded in such a migration, and is based on information and perspectives it shared with us. Email, Web conferencing and instant messaging migrations are always complex and cumbersome, but those that involve both a new vendor and a change in the provisioning model (from on-premises deployment to the cloud) are especially challenging. If we add in the complications of 346 separate worksites in 55 countries around the globe and migration to a new and unproven cloud platform, we are approaching a "perfect storm" of complexity.

This was the situation Flowserve (a supplier of pumps, valves, seals, automation and services to the power, oil, gas and chemical industries) found itself in 2010. Based in Irving, Texas, the company had been running IBM Lotus Notes/Domino for 10 years, but the aging system was cumbersome to manage and the company determined that there was little value in running its own email infrastructure. In early 2010, Flowserve made two critical decisions — to move away from Notes/Domino and to engage a third-party to run its email service.

Consequently, in the first quarter of 2012, Flowserve successfully moved nearly 12,500 users to the Exchange and Lync components of Office 365. The wildly complex migration was challenging and not without hiccups. Here's how it did it.

Brushes With BPOS

In early 2010, Microsoft was marketing its first multitenant email cloud service, called Business Productivity Online Standard Suite (BPOS). Microsoft offered a very low introductory rate to Flowserve, as one of the first large companies to move to BPOS. Microsoft also promised a very low fee for migrating users directly from Domino to the Microsoft cloud. Flowserve signed the contract in June 2010. However, further investigation of BPOS revealed shortcomings that called into question is suitability for a large, distributed organization like Flowserve.

The company found that BPOS:

• Lacked a single sign-on mechanism for users.
• Required service requests for simple operations, such as activating ActiveSync.
• Had an immature support infrastructure.

Office 365 Arrives

During this time (it was now late 2010), Microsoft began talking about the follow-on release to BPOS, which represented the move from the 2007 wave of Microsoft products to the 2010 versions (Exchange 2010, for example). Given the shortcomings of BPOS and the promise of what became Office 365, Flowserve decided to wait for the new release, which Microsoft moved into production on 28 June 2011.

Hiring a Consultant

Flowserve hired Slalom Consulting to lead the company through the technical migration. Microsoft was a supportive partner during the process.

Dual-Hop Migration

One of the first things Slalom suggested was a move from Domino to a premises-based version of Exchange 2010 (using its own Domino to Exchange migration tool).

These Exchange 2010 mailboxes would then be migrated to Office 365 cloud — a dual-step migration that required additional effort, but delivered several key advantages, such as:

• The ability to identify and rectify issues prior to the end user using Exchange for the first time, such as Active Directory attribute errors, corrupt Lotus Notes mail files, mismatches in Active Directory and users with duplicate Active Directory accounts.
• The ability to perform large volumes of conversions (from 50GB to 150GB per night) weeks in advance, allowing Flowserve to simply convert the small amount of changed data (the delta) on the night of the production migration.
• Allowing Flowserve to have much greater control over the timing of the migration. The company was keen to have all conversions done by 08:00 hours local time, so it had to ensure the migration was not still in process when the end users arrived in the morning.

Preparing the Battlefield

To migrate users effectively, Flowserve needed detailed information on all device configurations, including operating system and version, memory and disk configuration, version of Notes and site location. Flowserve was able to collect that data with a sophisticated, custom-built Notes application, which was supplemented by data from a Symantec Altiris asset management system. The data was needed to coordinate an effective migration. Without it, Flowserve would have faced long delays while it gathered that information, as other organizations have found.

Prior to the migration, Flowserve had moved to a standardized PC build, which would facilitate its goal of a "minimal-touch Day 1" experience for the user. The global build included:

• Internet Explorer 8.
• Office Professional 2010. This was also a "follow the sun" rollout that occurred prior to the migration.
• Remediation of operating system environmental issues, which were detected in the inventory assessment described earlier.
• The Outlook Day 1 package consisting of an autodiscovery utility, Lotus Notes client fixes (including a move to Notes basic mode, which uses far less PC processing power, compared with the rich Notes client) and the Microsoft Lync instant messaging client.

All the software was distributed before the migration and executed to predefined schedules. Flowserve upgraded approximately 100 64-bit Windows XP machines to Windows 7 64-bit, due to Office Professional 2010 not being compatible with 64-bit Windows XP.

Mailbox Migration Details

The initial data load required for conversion from Domino to Exchange put substantial stress on the network, but the presence of Riverbed WAN acceleration and caching services helped considerably during the initial population of Outlook 2010 on users' PCs.

The sequence of the data migration was as follows:

• Calendar, contact and email data was migrated approximately 21 days before the final conversion in background mode, unbeknownst to the user (from Domino server to Exchange server at a data center in Gilbert, Arizona). This pre-migration gave the team the opportunity to identify issues (such as corruption) that might cause problems. Overall, between 15% and 20% of Domino mailboxes needed some remediation as they were converted to Exchange.
• The night before a user was migrated, only the changes made to the mailbox since the initial conversion were required to be moved. With this approach, the team stayed 21 days in front of the migration. Final migrations kicked off at 17:01 hours (local time). The delta synchronization was typically less than 10% of the total mailbox volume. Mailboxes ranged in size from 300MB to 6GB, with the average being 700MB.
• When users who arrived at work at 08:00 hours successfully launched Outlook, the client pulled down the entire mail file to their PCs. This is where Riverbed played a key role in limiting the amount of "true" data moving across the remote site's small WAN circuit.
• Approximately one week after the conversion to Exchange 2010, these users were migrated to the cloud. This staged migration supplied Flowserve with the control needed for an orderly migration.

Flowserve migrated between 1,200 and 2,100 users per week over eight weeks. The company established a global schedule and migration at each site, starting at 17:01 hours (local time) to gain maximum migration time and minimize user disruption. The large variability in the number of mailboxes moved per week was because business managers played a key role in defining migration dates.

End-User Communication, Support and Training

End-user communication, support and training were a huge part of the migration, so Flowserve hired a full-time consultant to coordinate and conduct training.

Highlights of the program included:

• Development of "a day in the life of an Outlook user" training material packaged as a Day 1 survival guide. It included common actions to carry out in Outlook on the first day, such as adding signatures and creating rules.
• Flowserve developed a comprehensive Domino and Exchange migration/coexistence guide, since many common functions would not work, or were compromised, during the migration, such as calendar invitations, calendar rescheduling and mail routing.
• Flowserve also offered training on Outlook, which was more detailed than the information in the Day 1 survival guide. It offered regional training webinars three times a week (in the early morning to serve users in Europe, the Middle East and Africa and Latin America, at mid-day for North America and late at night for Asia/Pacific). Overall, approximately 35% to 40% of users took the non-mandatory training. Some sites had 100% participation as the result of attendance contests that rewarded winners with free lunches.

The company had one full-time equivalent contact, dedicated to communications with users.

Highlights of the communication program included:

• Each user received eight email messages, each with a specific purpose. Some messages included migration instructions.
• Communications were in eight languages.
• Communications were addressed to end users by name and some messages copied the associated "site champion," general manager or divisional IT staff.
• Creation of a Microsoft transition "portal" to post training schedules, FAQs, migration schedules and more.

Flowserve also invested heavily in supporting users. Even though Flowserve has a help desk, it also hired an external help desk provider to take Level 1 calls on the day of the migration. Callers who had just been migrated were routed directly to the help desk provider. Flowserve did not ask end users to call a new number and did not change the support process in any way. The automated answer system was customized to route users seamlessly to the outside provider, and this triage mechanism worked very well.

At locations without IT personnel, Flowserve enlisted a site champion to act as an IT liaison contact and promote the migration locally. The site champion was typically someone established at the site for a long time and instrumental in ensuring local compliance and awareness and who could ensure appropriate language translation.

Top Migration Issues

Throughout the project, the Microsoft account team was helpful and Flowserve also received help from its Microsoft technical account manager. Flowserve did not buy any additional support from Microsoft for the migration, but did establish weekly meetings with higher-level Microsoft cloud executives who could expedite thorny issues that were proving difficult to resolve.

Apart from some initial issues with the Microsoft account team, Flowserve believed that Microsoft was fully vested in the migration and worked hard to make it successful.

The top three issues encountered during the migration were:

• Stabilizing the Active Directory Federation Service and directory synchronization processes. Flowserve found a bug in the way the cloud authenticated Lync users, which led to many Active Directory lockouts.
• The inability to predict how much data Microsoft could upload in any given night.
• Flowserve would experience failed migrations when Microsoft moved users from data center to data center during the migration, since some migration services were hardwired to work with a specific site. In other cases, users had hard-coded server names for ActiveSync support, causing the mobile service to fail. At other times, the tenancy relocation forced users to re-authenticate.

Mobile Strategies

During and after the migration, Flowserve had all 2,500 or so of its Research In Motion (RIM) BlackBerry users continue to use its on-premises BlackBerry Enterprise Servers because the RIM-supplied Office 365 BlackBerry service was still in beta. Since the service went live, Flowserve has completed the move to the RIM cloud. The migration to the no-fee RIM Office 365 cloud has been difficult to manage because users often try to activate their new BlackBerry account before the migration is completed, thereby causing migration failure. This is not RIM's fault, but reflects the inability to forecast how many mailbox migrations can occur on a given night due to multiple companies contending for the same bandwidth while onboarding the same tenant.

Flowserve now allows users to use Apple iPhone and Microsoft Windows 7 phones, and the ActiveSync devices have had a smoother technical deployment.

Networking Impact

Previously, Flowserve had run Domino over its WAN, but Office 365 traffic runs over the Internet, so some networking adjustments had to be made, such as:

• Adding more Internet points of presence around the globe.
• Ensuring the destination Office 365 IP addresses were correctly classified in class-of-service (CoS) configurations. The CoS specified that Outlook and Lync should not impede voice-over-IP and Oracle applications, and that Outlook and Lync traffic was a higher priority than normal Web page traffic.

There was initial concern over the impact on bandwidth — but aside from the day-one load, there was no impact.

End-User Feedback

On the whole, user feedback has been positive. Users enjoy using email without a VPN and have embraced Lync for Web and video conferencing, as well as for PC-to-PC, voice and instant messaging. The enablement of Android and iOS devices is also popular, as is the Outlook client (rather than Notes).

Migration Resources

Implementation partner Slalom was engaged for about eight months with an average of three full-time equivalents.

Flowserve estimates that its own personnel contribution for the eight months was as follows:

• Full-time program lead
• Full-time project manager
• Full-time messaging lead
• Full-time trainer/change management coordinator
• Part-time Domino developer
• Full-time site champion coordinator
• Full-time communications resource
• Part-time infrastructure resource (mostly data center server builds)
• Part-time network analyst
• Part-time process engineering analyst
• Full-time Active Directory lead

There was also a monthly steering committee meeting. The committee was composed of the chief accountability officer and business and IT leaders.

Flowserve also held weekly meetings during the active migration. These averaged 25 attendees from Microsoft, the external help desk provider, Slalom, Flowserve and Unisys (Flowserve uses Unisys for Level 2 support).

Steady-State Operations

The experience with Office 365 in steady-state mode has been very good. For example:

• There have been no significant unplanned downtime events.
• The vast majority of support calls relate to the enablement of new users or features, rather than operational incidents.

Flowserve believes that managing Exchange and Lync in the cloud will require approximately one quarter of the technical personnel required to manage Domino email and Sametime instant messaging (one full-time person vs. four full-time people for the Lotus products).