Actifio Transforms Top 20 Global Bank
How one of the world’s largest banks created a “Database-as-a-Service Cloud” that accelerated application development cycle times while reducing costs
Executive Summary: Resolving the DevOps Dilemma
The world is changing fast. Like all global banks, this large financial institution needed to keep up, staying agile in the marketplace by developing new capabilities powered by innovative new software. To do that, the people developing this software needed fast and frequent access to the highest fidelity test data possible, to maintain quality standards and eliminate surprises when deploying to production.
As always, that data needed to be protected in line with compliance requirements, and the “always on” demands of the bank. The operations team needed to maintain tight control of critical database systems, each of which was bound to expensive and complex legacy infrastructure and processes.
Achieving this with infinite resources would be easy: Just throw more storage, software, and staff at the problem, and eventually it would go away. But the need to manage the conflicting demands of data access for Dev and data control for Ops without “breaking the bank” left the team searching for a better way.
That way was Actifio. The bank tested and deployed Actifio Copy Data Virtualization worldwide to protect and provide Oracle database provisioning as part of a new Database-as-a-Service cloud, improving compliance, recovery times, and the agility of their DevOps efforts. In the process, the global firm saved millions of dollars in infrastructure, software licensing and operational costs.
WHO: Global Consumer and Investment Bank
CHALLENGES: Slow restores of large databases violated SLAs; Need for faster provisioning of data for DevOps teams
IT ENVIRONMENT: Petabyte-scale Oracle RAC and ASM Database environment; Various OSes including AIX and Linux; EMC VMAX storage
SOLUTION: Actifio Copy Data Virtualization platform to protect and provision/recover databases instantly, automatically from a Database-as-a-Service Cloud.
• Over $25M in total financial benefits
• Over $1M in hard cost savings in the first year.
• 90% faster database provisioning for DevOps teams
• Recovery of terabyte-sized Oracle databases in minutes
One of the world’s Top 20 global banks, this financial institution has hundreds of millions of client accounts in more than 150 countries worldwide. It delivers corporations, consumers, and institutions a wide range of services and products, including investment and consumer banking, wealth management, securities and brokerage offerings.
The IT services to support the firm’s many business units and offerings include thousands of developers, DBAs and operations staff, 20 data centers, petabytes of data, and a myriad of legacy technology platforms, including a dozen operating systems.
Risks come in many forms for such global firms, and a lack of responsiveness and agility in the marketplace worried IT leadership at this bank, especially considering growing competition from nimbler SAAS-like FinTech startups. At the same time, maintaining stable, highly-available services is critical, if not mandated, by regulations and SLAs with internal business units.
In light of these risks, managing Oracle databases across a business of this scale was a problem. With over 7 Petabytes of Oracle data supporting critical applications across thousands of databases on various Unix and Linux OSes, protecting and assuring access to Oracle databases for application testing and development were both priorities for the bank’s data team. The pressure to release new software capabilities to compete in the digital marketplace had grown more intense, so enabling development speed and agility was paramount.
On the operations side, one of the most pressing issues was the slow restore times for large databases—those over 1 Terabyte in size. It could take more than 24 hours to restore one using the bank’s standard Veritas NetBackup tool, and could involve up to seven different staffers because of the processes involved. Such recovery times simply didn’t meet internal SLAs, and created compliance risks. Both the operations and database teams knew it.
At the same time, the bank was also burdened by storing about 10 extra non-production copies of their Oracle databases as part of their application development lifecycle, which resulted in huge storage and management overhead. The management team realized this level of capital expenditure was not economically sustainable, nor was the operational expense to maintain these database copies. The staff time and effort to refresh non-production test and development environments with updates were onerous and could take up to three weeks (if the Oracle database was recovered from a tape backup), or up to ten days (from a disk backup system), and involve a 12-step approval and provisioning process touching five different teams. Needless to say, this infrastructure-bound process did not support the goals of DevOps agility.
Proving Backup and Recovery Speed
The bank’s engineering team first engaged with Actifio to tackle its burdensome (and SLA-defying) Oracle database restoration problem. The bank’s IT executive team understood how fundamentally different Actifio’s Copy Data Virtualization platform was from legacy backup products, but they and their database administration and the operations teams had to test the platform, and frankly ‘see it to believe it’ on a real, important application.
As a result, the bank conducted a thorough Proof-of-Concept on an invoicing application that was housed primarily in the United States but supported global operations. Actifio appliances were deployed alongside the production systems, used RMAN APIs to do an initial ingest (full backup) of the Oracle databases running in a RAC configuration with ASM, and then subsequently captured only the changed blocks. This approach drastically reduced the backup windows for the databases.
More important though were the recovery speeds. Because the Actifio appliances kept the Oracle databases in their native application format, and then simply presented (instantly mounted) virtual databases (synthetic full point-in-time copies) back to hosts for restorations, the database and ops teams could now recover their databases in only minutes. This contrasted with the Veritas approach (also used by other backup tools) of copying and physically moving the data blocks to be recovered from the backup server, and in the process, converting the databases from Veritas’ proprietary backup format to something usable by the application.
Database Virtualization for Database-as-a-Service Cloud
As the PoC was coming to a close, however, the bank’s IT executive leadership team was also solidifying plans for a new internal private cloud to provide much greater automation and agility in its infrastructure. While the bank was not able to leverage public cloud IAAS, it still benchmarked its infrastructure services against Amazon Web Services in terms of performance, cost and flexibility. Its internal private cloud was specifically aimed at making IT infrastructure and data services more responsive to its application teams and ultimately the business.
A critical part of this was establishing Database-as-a-Service infrastructure that would allow it to transition thousands of Unixbased Oracle databases to a cloud hosting Oracle databases on Linux VMs. This would drastically reduce compute, and other infrastructure costs, in addition to providing operational efficiencies.
After the Actifio backup and recovery PoC proved the utility of data virtualization technology to provide instant data access, it was clear that Actifio should also be considered as the data protection and now database virtualization engine for the new Database-as-a-Service (DBAAS) cloud.
The bank also evaluated another database virtualization product, which it had used in a departmental development project. Similarly to Actifio, the tool created a master copy of Oracle databases, and then provisioned virtual copies of it for test/dev teams to access.
However, there were a number of critical technical factors that led the bank to select Actifio as its global data virtualization standard for its DBAAS platform. Most importantly was Actifio’s ability to fully support Oracle ASM, including support for native transparent migration from staging to production environments. The other database virtualization tool lacked ASM support, which the bank deemed a critical element for connecting to production databases. Additional factors also set Actifio apart in the eyes of the bank’s technical teams: robust, bandwidth-efficient replication capabilities, integration with native Oracle technologies such as RMAN, and the ability to support multiple use cases (including backup) at enterprise scale across multiple parts of the application stack (VM, OS, etc. in addition to the databases).
Of course, the technical validation wasn’t enough—an extensive financial analysis was mandated by the bank’s leadership and procurement teams. Actifio’s consulting division engaged closely with the bank at all levels to collect information on their current state architectures, processes and costs, and created a business case with the bank’s data and assumptions to assess the realistic business impact of rolling out Actifio as the database protection and provisioning engine of the global DBAAS project. The financial impact was measurable, risk-adjusted, and huge —exceeding $25 million over the life of the system. With such a clear positive business impact, the bank’s CIO, finance, and executive stakeholders gave a greenlight to the project.
After selecting Actifio, the bank began deployment on two fronts: Customization of a self-service portal front-end, and a global Actifio appliance rollout. The portal leveraged Actifio’s RESTful APIs so that any development, test, or DBA team member with the right credentials can access appropriate databases to work with, on-demand, and on-schedule. More specifically, staffers with the appropriate privileges are provisioned masked, read-write, virtual copies of databases, so they can perform recoveries, application testing, or development, integrity checks or any other task necessary for development or operations work.
The global roll-out of Actifio included the deployment of dozens of Actifio CDS hardware appliances across each of the bank’s major data centers in the world: Six in North America, one in the UK, one in Germany, and three across Asia. After approximately one year of working with Actifio, the bank transitioned to deploying Actifio Sky virtual appliances on VMware ESX. Both appliances use Actifio’s integrations with Oracle RMAN, RAC and ASM to deliver the same data virtualization capabilities.
The DBAAS Cloud now supports over 2,000 of the firm’s Oracle databases, with a new database being added to the platform every 90 seconds, on average. Once on-boarded to the new Actifio-based DBAAS platform, databases are no longer backed up by the legacy backup systems, dramatically improving availability SLA’s while dramatically reducing costs related to licensing, infrastructure, and operational complexity
The business will recover the fully-loaded cost ahead of schedule:
- A confirmed $1M hard-cost capex savings in year one.
- Only six months to achieve cash-flow positive benefits.
Over the five year life of the project, major cost savings will be achieved:
- $25 million in savings from storage consolidation and data management improvements per petabyte of the bank’s Oracle environment.
- $19M+ of that is from hard cost savings, derived from reducing software license and maintenance costs, and eliminating copies of databases. The reduction of copies resulted in a 55% storage savings / reclamation.
- $5M+ in soft cost savings, from operational improvements.
- In year one alone, an estimated $20M+ in other infrastructure savings from the overall DBAAS cloud project and Unix-Linux migration—aside from Actifio-derived storage savings.
- Rapid, automated database restoration; multi-terabyte Oracle databases can be recovered in minutes, by a single requestor.
- A 90% reduction in database provisioning time, for DevOps activities, reducing developer and DBA wait-time.
- Self-service automated data provisioning, reducing friction between operations and dev/test teams.
- Elimination of 55% of storage associated with each Oracle database for operational and development use cases.
- Automated integration of data masking, providing improved data control, reducing the risk of data breaches, without sacrificing agility.