GAIN Capital

GAIN Capital is a US-based provider of online trading services, headquartered in Bedminster, New Jersey. The company provides market access and trade execution services in foreign exchange, contracts for difference and exchange-based products to retail investors.....

Introduction

GAIN Capital existing Comms Room was based within an office environment. Their requirement was to relocate their equipment to a new location due to the expiration of their lease. GAIN chose to take advantage of this opportunity by migrating their Comms equipment to a managed service environment, as they already had existing infrastructure at Interxion Data Centre. GAIN also wanted to reduce their physical infrastructure footprint before migrating. In order to reduce the footprint prior to the physical migration, it was a requirement to perform virtual migrations.

The Solution

GAIN Capital Data Centre Audit, Environment Readiness and Relocation Services

Technimove Professional Services team were engaged by GAIN Capital to lead and develop the migration programme. The programme consisted of a discovery phase, environment readiness, virtual/physical migration delivery and dilapidation of all decommissioned hardware.

1.

Discovery Phase

A full low-level, detailed Audit and Cable Trace of GAIN’s current environment, across 2 sites (Park House & Interxion in London). This enabled the Technimove team to provide GAIN with a snapshot of their current setup.

2.

Environment Readiness

To create new rack elevation designs, provide information to Interxion regarding additional power requirements needed to support the migration, patching schedule consultancy and verification, new network refresh install and to pre-cable GAIN’s racks at Interxion ahead of the physical migration.

3.

Virtual/Migration Delivery

Planning and execution of test Veeam implementation, P2V – to migrate physical to new virtual estate, V2V to migrate all 250 virtual servers onto the new VSAN platform and to migrate all remaining physical servers to Interxion.

4.

Dilapidation

To provide ITAD services to assist with the dilapidation at Park House, by de-racking all servers, storage, network equipment marked for disposal and removing all racks, structured cabling and network cabling from the comms room.

To ensure a smooth and successful delivery, Technimove undertook the following work packages:

Project Management
Physical Audit
Rack Design
Document Consultation and Verification
Network Install & Pre-Cable
Migration Planning
Veeam Implementation
Test Virtual Machine Migration
P2V Migration
V2V Migration and VSAN cluster testing
Physical server and storage (including 3PAR) Relocation
Physical Migration of hardware from Park House to Interxion data centre
Physical Migration of hardware from Park House to Devon House data centre
Decommission of hardware & disposal

Delivery

Detailed Phase-wise breakdown of tasks

1.

Project Management

Technimove provided a PRINCE2 accredited Project Manager, who would oversee the delivery of the statement of works. This PM was engaged in the Project Start Up phase, ahead of the Project Initiation. This was in place to understand the requirement at a virtual/physical infrastructure and service delivery level. The deliverables included a PID, Highlight Reports, RAID, Project Log, RAMS, in addition to change control management and the creation of a project plan.

From all the information collated from Project Start Up, Technimove were able to ensure they resourced and planned all pre-migration activities accordingly, which ultimately ensured the downtime and risk of the virtual and physical migrations were kept to a minimum. This saw Technimove manage and oversee the Veeam Implementation, Testing Virtual Machine Migration, performing the P2V, V2V and physical server/storage data migration.

2.

Planning Phases

Rack Design & Power Requirement

The first stage of the planning was for Technimove to review the PKH (Park House) Master Spreadsheet, for equipment marked required for migration from Park House to Interxion. Once reviewed, our internal technical team commenced the Rack Design. This included the aspects below:

Distribution of devices to the available “U” space
Planning the distribution to make logical sense for the Pre-Cable activity
Ensuring the rack depths were appropriate before housing any devices in the racks
Colour coding of devices by type
Inclusion of old and new host names against each device
Calculation of additional power requirement
Management of changes, via change control

Upon completion of the Rack Design, GAIN Capital were provided with the document for review. Once reviewed, accepted and signed off, Technimove set up meetings with Interxion Service Providers to explain the additional power required, and to work together to understand the best approach to implement.

As a result, it was agreed and confirmed by all stakeholders to replace the 16amp horizontal PDUs with two 32amp vertical PDUs.

As this was a live production environment, challenges were faced to ensure that no power outages occurred during the implementation. As single-phase devices already present within racks, it was not possible to bring down one PDU at a time. Taking this into consideration, it was decided to allocate a window of downtime to install new PDUs, across one weekend, which led to a successful installation.

The next stage of the planning was to commence the document consultation and verification, for the creation of Network to Network and Server to Network Patching Schedule documentation.

The Physical Audit, completed by Technimove during the discovery phase, was used to capture all servers, network devices and ports in scope to be migrated and was transposed into Technimove’s Patching Schedule Template. Technimove and GAIN’s technical teams worked together to produce the patching schedule.

The following actions, consultation and verification was completed:

Converting initial audit documentation for servers, network, storage and ports in scope, into patching schedule format
Advising GAIN Capital on physical port mapping from device to device
Translated GAIN Capital’s “B END” port allocations to servers into patching schedule
Sanity checking port conventions against physical device ports to ensure no mispatches during the Pre-Cable activities or Physical Migration
Assigning cable colours as per GAIN Capital agreed conventions i.e. Nic 1 – Blue, ILO – Grey etc…
Checking port orientation and advising on any issues with hot/cold aisle
Re-assigning ports, if requested, by GAIN Capital
Advising on specialist cables that needed to be procured
Manage all changes via change control

Upon completion of the above, a final onsite workshop was organised to run-through any last-minute changes, discrepancies and additions before implementing a change freeze ahead of the pre-network install and pre-cable service.

Document Consultation and Verification

Once the Rack Design and Patching Schedule documentation was completed and verified, a change freeze was implemented to enable Technimove to carry out the Network Install and Pre-Cable.

The Network Install was part of GAIN Capital’s network refresh, which involved collecting refurbished switches that were not being utilised at Park House. This was a good opportunity to pre-install switches ahead of the Physical Migration, as these switches had been pre-configured and can be tested ahead of the move itself, minimising downtime.

Along with the Network Install, Technimove completed a service to Pre-Cable all network cabling within racks. This was a vital part of the planning, as 65% of the cabling required routing in overhead cable trays. This was due to live equipment already present within the racks, meaning that top of rack switches were not always possible and patch panels were already fully utilised. Performing this ahead of the migration was key to minimising downtime.

Network install and pre-cable

The Technimove team performed the Pre-Cable from core switches, across rows where required, installing copper and fibre cables. All cables were labelled top and tail and managed to the correct U position in line with destination rack plans and patching schedules.

Migration Planning

Migration Planning was required at this stage to discuss the approach and agree how the Physical and Virtual Migrations would be completed.

It was agreed that the P2V and V2V would be migrated across five days, outside of business hours.

The Physical Migration was originally agreed to be completed across one weekend. However, since the original plan, Technimove and GAIN Capital were ultimately forced to postpone the migration, due to the COVID-19 pandemic which caused a lockdown across the UK. This caused a great constraint on the delivery of the project and there was pressure on GAIN Capital to vacate their office space in Park House, due to the expi’ring lease.

However, with our Project Management team working along side GAIN Capital’s Senior Management the solution was made by extending the lease, which ultimately led the re-planning of the migration date. To ensure that Technimove delivered the migration to Interxion in line with the government’s guidelines, Technimove extended the timeframe from a weekend to 4 days. This ensured that all personnel involved were working in a safe environment, whilst still achieving a successful delivery.

The planning work above involved using key product tools such as Project Plans, RAIDs and Weekly Reports. These tools and reports were reported up the chain to GAIN Capital’s board on a weekly basis.

3.

Delivery Stages

Veeam Implementation and Testing

Technimove’s first objectives were to implement and test Veeam platform. In order to deliver this, GAIN Capital supplied a 64-bit Windows 2016 R2 server environment as well as providing all IP addresses for Veeam, access to the VMware environment and administrator access to the Veeam server.

Technimove was responsible for the following actions to implement the Veeam platform:

Gather the Veeam licenses procured by Technimove
Install Veeam 9.X on the designated Veeam Server
Install all the available patches
Configure Veeam as per Veeam’s best practices
Configure Networking as per design on the Veeam server
Install and configure Veeam Backup and Replication Console
Install all the available patches
Group the VM based on the application relationship
Configure VM replication group
Configure Veeam proxy as per Veeam’s best practice
Configure networking for Veeam proxy to communicate to Veeam server
Configure replication of up to 4 test virtual machines between the existing production and the new virtual platform
Monitor the replication job

The 2nd objective was to test the Veeam Implementation, Technimove was responsible for the following actions to test Veeam:

Provision the virtual machines on the existing production and the new virtual environment for Veeam Proxy
Test the replication by failing over the virtual machines from the existing production to the new platform
Verify normal operation
Document the failover process

Once the implementation of the platform, testing and acceptance criteria was met, this enabled Technimove to use this platform to migrate the GAIN Capital’s virtual machines.

Physical to Virtual Migrations – P2V

Technimove were now at the stage of the project where the necessary pre-requisites have been completed and implemented. This meant the Migration work packages could now commence.

The first work package was to deliver the Physical to Virtual Migration (P2V). Technimove’s objective was to virtualise 5 physical servers on the new VSAN platform, and to ensure all VMs are migrated/accessible from new infrastructure.

In order to deliver the above, Technimove were responsible for the following actions:

Test local account/domain account access to the physical server which are in scope for migration.
Install the latest version of VMware Converter
Complete the VMs migration readiness
Migrate the servers to the new environment
Application shutdown (completed by Stakeholders)
Power off from the existing infrastructure
Power on the VM from new infrastructure and test the VM access
Install latest VMware tools and confirm that virtual machines hardware level is at a pre-agreed value during the time of migration

Virtual to Virtual Migrations – V2V

Stage 4 of the delivery was to complete the Virtual to Virtual Migrations (V2V). This involved migrating 250 virtual servers on the new VSAN platform, hosted across two SuperMicro devices.

GAIN Capital provided remote login details to Technimove. Once retrieved, the following actions were to be completed by Technimove:

Test local account/domain account access to the virtual server which are in scope for migration.
Application shutdown and complete power down of the VM’s from the existing environment
Final replication sync
Power on from for the new infrastructure
Test the VM access
Install latest VMware tools and confirm that virtual machine’s hardware level is at a pre-agreed value during the time of migration
Clean up the old drivers

During the course of the Migration, Technimove faced a series of challenges. The issues started to arise after completing 80 virtual migrations onto the new platform. Circa 20 VMs failed because of disk failure errors. Due to ongoing issues, a rollback had to be invoked. Technimove and GAIN Capital commenced the troubleshooting to discover the root of the problem. It became apparent that GAIN Capital received a faulty batch of hard drives, which ultimately caused the virtual migrations to fail. As a result of faulty drives, this also initiated software problems on the SuperMicro devices.

There was a 4-week lead time to retrieve replacement hard drives and to fix the software issues. However, because of the time constraints of the project it was decided to approach the re-seller to obtain two new units, as they could be delivered within a week.

Once the new units were delivered, it took a weekend to configure and prepare the devices. Upon completion of the preparation work, it was decided to leave the units to run for a further two days to ensure no software or hardware errors re-occurred.

Completing the above steps meant that the remaining 170 virtual migrations could now re-commence and were successfully completed across 4 days.

Physical Server and Storage Migration to Interxion

Upon completion of all virtual migrations, the project was now approaching its final stages.

Technimove’s objective, was to relocate the reduced footprint of GAIN Capital’s infrastructure of 90 devices, from Park House to Interxion Data Centre. However, due to the COVID-19 pandemic and ensuing government-imposed lockdown, the migration was postponed.

Once the lockdown measures were eased, Technimove and GAIN Capital could re-schedule the migration. Technimove’s Project Manager had scheduled the Migration to be completed across 4 days, Friday to Monday. Tuesday was added within the migration plan as a contingency measure. This was in order to follow the government’s guidelines whilst keeping all onsite personnel safe.

Technimove’s scope was to perform the following:

Pre/post-checks and power off/on all HPE 3PAR equipment
Shutdown/power on all server and storage equipment, as per shutdown scripts and power down/up sequence
Provide a network resource onsite to assist GAIN Capital during network testing
De-Cable, De-Rack, package equipment, transport, Re-Rack and Re-Cable all 90 devices in scope.

On the first day of the Migration, Technimove allocated two teams; a power down and migration team. The power down team was first to arrive onsite, they had an 8 hour window to gracefully shutdown all servers, storage and 3PAR devices.

The migration team arrived 1 hour prior to the completion of shutdown. Upon the migration team’s arrival, engineers began by preparing all packaging material required for the Migration. The migration team lead then liaised with the team lead in charge of the shutdown to ensure completion, ahead of commencing work. Once confirmed, the migration team commenced the decommission and labelling of all devices required to be migrated in line with the equipment list.

Equipment was de-racked before being individually wrapped in anti-static bubble wrap and placed into purpose built, foam lined, server flight cases.

All rail kits associated with the migration were labelled in line with the destination rack plans before being de-mounted and placed into purpose built, foam lined, rail kit flight cases.

Upon completion of the de-installation, Technimove’s team lead performed a walk round with the customer along with an equipment list, to perform final checks that all devices in scope have been removed.

Flight cases were secured and security sealed in Technimove’s Mercedes GPS Tracked, solid sided vehicles.

The pre-fuelled vehicles travelled directly to the destination site at Interxion, where the Technimove team unloaded flight cases and moved them into position within the new facility.

Technimove commenced breaking the security tags and seals. All rail kits were unpacked and installed as per the rack designs. After rail kits were installed and flight cases were brought into the DC room, the team reached a milestone and then ceased works. The team ensured all devices were secured within GAIN Capital’s DC environment, signed off and left site until the next day.

On the second day the install re-commenced, the migration team signed in at Interxion, gained access into GAIN Capital’s DC hall and commenced the re-racking of devices as per the rack designs.

The challenge for the re-racking was due to the government COVID-19 guidelines. Technimove had a reduced team onsite and there were 5 HP C7000 devices to re-rack mid height. To assist the team with this challenge, Technimove utilised mechanical Genie Lifts, which enabled a safe install.

The other challenge the team had were that a percentage of the remaining server equipment were originally racked on faulty or missing rails. Technimove bring 0u shelves on every migration as a best practice measure for this type of occasion. This enabled the team to install the devices without rail kits successfully.

During these works, Technimove had supplied a network resource onsite at Interxion. The engineer assisted GAIN Capital to test the network.

Upon completion of the re-racking, the migration team hit the milestone required for this day. The team lead signed off with the customer and left site.

On the third day the re-commissioning work commenced. All devices were patched in line with the patching schedule. There were circa 550 x network cables to patch. It was imperative that the patching was completed with accuracy. To ensure there were no mispatches, the team would work in pairs to patch, review and sanity check the cabling.

The team had reached the third day’s milestone. Upon completion the team lead notified the client and Technimove’s Project Manager of work completed and signed off.

The migration was now on the final fourth day. Technimove had a power up team attend site along with our Post- Commissioning team, who completed the physical migration.

Technimove followed a power up sequence and script for all servers. However, there was an issue with the network connectivity and the credentials for logins would not work. There was also a fault with one of the ILO management ports on a C7000 device. After troubleshooting with GAIN Capital regarding the network connectivity and login, it was decided to reset the credentials and re-start the configuration process again. The C7000 was under warranty, so HP sent out an engineer within the same day to fix the faulty ILO management port.

The above issues caused a 6 hour delay, which meant that the fifth contingency day included would be needed. However, due to Technimove’s flexibility, additional resources were allocated on the fourth day to power up all critical services ahead of the fifth day to give re-assurance.

This meant minimal work was left on the contingency day, all remaining servers were powered on by Technimove. Upon completion, the team leader reported back to the client and Technimove’s Project Manager, before providing a sign off of the service.

Dilapidation

Technimove’s final stage was to provide ITAD services to assist with the dilapidation of the Park House comms room. Technimove removed all server, storage network devices marked for De-commission, and the associated racks and cabling.

Once all items had been received by the recycling warehouse and given their unique reference number, the disposal team audited each item and recorded their details (make, model, serial number, asset number and unique reference number) on an electronic report. Once all the items’ details had been recorded, the team immediately removed any magnetic or solid-state hard drives and checked for any other data carrying media. The media then entered Technimove’s data suite for sanitisation as per our data capability statement.

Technimove offered two levels of software overwriting solutions. Hard Disk Manager Software can be used, which meets US DoD 5220.22-M overwriting standards or industry leading CESG certified (on magnetic hard drives) Blancco Software.

All hard drives that were collected are wiped as standard, whether Technimove are told if they contain data or not. The official standard to which Technimove wipe hard drives is ‘British HMG Infosec Standard No: 5 (Enhanced)’.

During the hard drive overwriting process, both software solutions kept a log of start time of the write, along with the date, size of hard drive, serial number of hard drive and completion time and date. The software verified the wipe and the method used and this will be stated in a report at the end of the wipe. In the extremely rare event that a hard drive failed to wipe correctly, this hard drive was then physically destroyed.

Once all items had been fully disposed of, a certificate was raised which stated the following information:

Address collected from
Date items were collected
Address items were items were disposed at
Date items were disposed of
Number of hard drives destroyed (if any) and whether this was on or offsite
Full audit of the equipment including make, model, serial number, asset number and number of hard drives.

This certificate, along with the collection note, hard drive wiping report (where applicable) and waste transfer note was then sent to the nominated project leader. All documents will be kept on record for future reference for up to 3 years. Should the client require copies of any documentation, a written request is to be sent to Technimove.

The Result

Technimove completed the full scope of work, as planned, and with excellent feedback from the client, successfully reducing GAIN Capital’s infrastructure footprint from an initial 243 devices, to only 90 devices. The team completed the Physical Migration from Park House into Interxion Data Centre, ahead of schedule, ultimately providing GAIN Capital with a constant managed service environment.

Feedback

The customer provided the following feedback:

“We’re very happy, overall, it was really good! Everyone found your team to be engaged and flexible, even helping us deal with our own self-inflicted issues with the move. Thanks for all your help and the team, great work done over the course of the project despite the challenges!”