How CIOs Are Driving IT Infrastructure Innovation

Washington Mutual's failure last September marked a precipitous moment of this recession. But for JPMorgan Chase, it was an opportunity. And for Guy Chiarello, JPMorgan Chase's CIO, it was the right time to invest in the company's IT infrastructure, including millions of dollars in new mainframe computers. By acquiring a big chunk of WaMu's business for $1.9 billion, including its prized West Coast branches and its pile of bad home loans, JPMorgan Chase already has boosted its revenue in retail banking and consumer lending. It upgraded to IBM's Z10 systems shortly after the acquisition, creating a mainframe infrastructure that can handle current volume and create capacity on demand so that the company could more quickly absorb WaMu's portfolio and customer data and have more efficient capacity for when business picks up. While some CIOs have put the brakes on IT infrastructure initiatives, Chiarello's team has been busy investing time and money in hardware, refreshing and in some cases virtualizing desktops and servers throughout the company. Upper-hand negotiations with vendors are providing what he estimates are 25% savings over what he would've paid for new hardware during better times, setting price points that will be hard for vendors to ratchet back up when the economy turns around. Bad times? For Chiarello, these are good times to get work done, even with the "tremendous pressure" on his IT budget. Chiarello doesn't know how long the vendor price competition of this economy will last, but the company's positioned "to make it last forever, because I won't pay a dollar more [than I paid] for the last thing I bought." Call it IT infrastructure innovation, and it's surging along at many companies. Yes, these are undeniably tough times, with revenue shortfalls slamming IT budgets and forcing layoffs that have IT unemployment back to 2004 levels, above 5%. Gartner predicts global spending on IT hardware will decline 16% this year, then be flat next year. It predicts IT services spending will decline 5.6% for the year, reflecting the lack of new projects in the pipeline. But to sum this up as "not much happening in IT" is a superficial view. Underneath the gloom there's a flurry of activity, new thinking, and even selective investments, as some CIOs seize the opportunity to both modernize and cut costs from their infrastructures before the upturn, when growth projects will again dominate the business technology agenda. Some of the most clever and innovative thinking in IT today has nothing to do with, say, software that helps grow the customer base or improve collaboration with partners. Because infrastructure innovations are all about making IT work better and more cheaply, they can gain support from business leaders in this economic climate. At the same time, some of a company's top IT stars, previously focused on growth-oriented applications, have become available to help find ways to optimize the most expensive systems and software. "When business is flourishing and demands are increasing, infrastructure has to serve the business in a progressive way," leaving less time in the portfolio to focus on infrastructure improvements, Chiarello says. "If you're in the CIO seat, the most exciting times for infrastructure are in down environments, and the most exciting times for application development are in the up times." Among the infrastructure innovations gaining momentum: server and, to a lesser extent, desktop virtualization; cloud computing, at least in pockets; open source software alternatives; new and more energy-efficient data centers; lower-cost voice networks, including unified communications; expanded IT automation; and new processes and software for tracking IT system costs and usage. In an April report, Gartner cites IT cost reduction as the No. 2 strategy for CIOs in 2009, a leap from No. 12 in 2007. Delivering projects that enable business growth dropped from the No. 1 strategy in 2007 to No. 3 this year. No. 1 is linking business and IT strategies and plans, up from No. 2 in 2007. After The Hyper-Caffeinated Growth JPMorgan Chase, which has repaid its $25 billion TARP funding to the U.S. government and posted a rising profit last quarter of $2.7 billion, is in a better position than many companies to invest in its IT infrastructure. But even companies whose growth is slowing or declining are using this time to fix what had gone wrong with their IT infrastructures. John Shepard, director of global IT infrastructure at Starbucks, joined the coffee-store chain in November 2006, back when a whole lot more people could plunk down $100 a month on morning espressos and pick-me-up afternoon lattes. "One thing we were good at was opening stores, seven or eight stores every day, all year long," Shepard recalls. "We relied on individual business units to do whatever they had to do to get business done." Efficiency took a backseat to growth, and as a result, Shepard says, IT infrastructure investment "was kind of a free-for-all." Just as Starbucks' success was a symbol of the good times, its declining earnings in 2008 and decision to slow down openings and close some stores were signs of an economy headed south. Talk intensified at the company about cost savings, and layoffs hit IT and other parts of the business by summer. In May 2008, Starbucks named a new CIO, Stephen Gillett, formerly CIO of Corbis, a digital image company owned by Bill Gates. By October 2008, Starbucks had embarked on a new central asset management approach to IT, which included a plan to track down every PC across all the business units. "The economic downturn was the catalyst that allowed us to come in and say, 'OK, guys, this is a behavior change that we have to go through together. If we can better the company and make some decisions together, we can come out of this a little better,'" Shepard says. Starbucks discovered close to 2,000 PCs "in drawers and cabinets" that no one was using. "We were spending money purchasing new PCs, and then we'd discover we had this huge inventory of in-warranty machines," Shepard says. Using Apptio asset management software, IT now tracks PCs, their depreciation, and warranties, and also identifies pricey software licenses no longer needed for some machines. Starbucks also is using the Apptio software to compare the cost of maintaining aging PCs to determine whether it's better in some cases to buy new ones, examining factors such as the IT support ticket history of individual machines and industry research on the typical life cycle of a specific model. These types of details can go a long way in helping Starbucks' IT department understand the condition and expected life span of any PC in the field without having to physically touch it. Sometimes the conclusion is that it's cheaper to buy a new PC. But overall, Starbucks is spending less on computers--avoiding $1.5 million of the $2 million it would have spent on PC purchases, Shepard says, mainly by redeploying unused systems. The company is now looking to extend its asset management effort to network printers, cell phones, and even servers. Advice From JPMorgan's CIO Given the information-intensive nature of its global business, JPMorgan's infrastructure innovation is far more complex than Starbucks' and most others'. Financial services companies typically spend 8% to 12% of revenue on IT--that would put the firm's IT spending at more than $7 billion this year. By comparison, retailers typically spend no more than a few percent of revenue on IT. "Financial services can be thought of as a technology business with a financial domain of expertise," Chiarello says. He ticks off all the economic downturns in his IT career--'87, '91, '01, and '08--as times he put his foot "all the way down on the pedal" on IT infrastructure improvements and innovation. "My whole shtick is offense versus defense," he says. "You can't save your way to success." After three decades in the pressure cooker of Wall Street IT, here are a few of Chiarello's insights into how to take the offensive in a down economy. Some of the best deals can be had with small IT vendors struggling to remain relevant in a consolidating market. Price savings are easier to negotiate for hardware than software. Meantime, the software companies will come after you, so get ready. "Software vendors lock down on maintenance revenue and do audits on inventory, trying to squeeze out whatever dollars they can," he says. His advice: "Get your asset management inventories structured well and understand the contracts, so you can go on the offense on that front and combat that activity." Chiarello has also spent a lot of time examining the "capital-versus-people" trade-off, looking for instances where it will be less expensive to upgrade to modern or commodity technologies than to have IT personnel keep older systems running. Often, saving money takes significantly different infrastructure strategies, and sometimes emerging technology. In recent months, JPMorgan has invested in IP networks to trim big long-distance phone bills in certain regions. His team has been examining every data center, assessing the cost per square foot, to ensure each one is getting optimal workloads, and bringing in VMware's server virtualization where it makes sense. And the company is building private cloud computing environments to run market simulations and risk calculations, and he's looking into using public cloud services. Chiarello also has close to 20% of employees--around 30,000 people--using virtual desktops, and he plans to expand that number. Since more of a virtualized desktop's computing is done by server-based software, he sees the potential to stretch PC lives from about three years to five or six. Chiarello's ultimate goal is to let employees build their own environments on virtualized PCs, tapping into only the applications they use. This approach, in turn, should give him bargaining power with software companies to pay for only the software the company uses. JPMorgan is also investing in more software-as-a-service applications, including those from Salesforce.com. What Would MacGyver Do? James Sims, CIO of Save Mart Supermarkets, is in a much different boat. With about $5 billion in annual revenue and an IT budget already less than 1% of that number, he was asked to cut $6 million from his budget this year. He's almost there. But Sims is used to living lean. "I've never found a company more scrupulous about the value of IT than we are," Sims says. "We're looking at everything, and questioning what does it really do. Does it provide value, and is that the only way we can solve that?" Sims wanted to celebrate the innovation that goes into cutting IT infrastructure costs, so he created the "Miraculous MacGyver Awards," silver trophy cups that riff on the 1980s TV show about a secret agent who pulled off heroics with everyday items, like plugging a sulfuric-acid leak using chocolate bars. Ideas for using low-cost, open source software have scored some MacGyver awards. Save Mart now uses software from the open source Hobbit project, for example, to manage and monitor some systems. Save Mart is replacing Microsoft SQL Server and Oracle databases in several instances with the lower-cost open source Ingres database. "We've really had our fill of Oracle's lack of partnership," Sims says, citing what he says is the vendor's unwillingness to negotiate on licenses or cancel licenses for systems Save Mart no longer uses. Using virtualization, Save Mart is running about 370 Linux servers, and the company recently invested in shared storage systems, eliminating miles of tape drives. Three years ago, he says, each Save Mart store had four individual servers running either Windows, AIX, or SCO. Using VMware, each store now has two servers running SUSE Linux that mirror each other, and two live, identical point-of-sales systems. In the case of a system failure, there's always a backup. "The old model would have been to get a technician out to the store," he says. That model became too pricey and time consuming, particularly after Save Mart acquired Albertson's Northern California division in 2007 and became a chain of 245 stores. The trade-off with Linux, Sims says, is hard work. "Instead of having a vendor with a turnkey solution and knowledgeable professional services, you have to download it and figure it out," he says. "That may not suit everyone, but it suits our style." Project S, And Other Tough Steps Like a lot of hospitals, New Hanover Regional Medical Center in North Carolina has been battered by the rising ranks of unemployed and uninsured patients, a drop in patient spending for discretionary medical procedures, government cuts to Medicaid and Medicare, pullbacks in philanthropy, and losses on its investment assets. CIO Avery Cloud's annual IT capital budget has been slashed from about $11 million to $9 million. The silver lining: "We get to spend time during this slower period to prepare the IT infrastructure to support strategic initiatives that will certainly come our way in the future," Cloud says. Enter "Project S"--for stabilizing systems and services. The hospital cut growth-oriented strategic IT projects by 70%, with the resources that remain moving to Project S. So far, New Hanover has avoided layoffs by moving people to the stabilization effort. And it has bought software from Compuware to manage IT Infrastructure performance, as well as the vendor's IT portfolio management software to ensure money is being spent on the right projects. New Hanover Regional is also investing in server virtualization to cut system costs. "I might not have had as big of an opportunity to do these things before, because the focus was on growth-oriented projects," Cloud says. These are also tough times for direct-mail company Valassis, which, among other things, makes coupon booklets for newspapers, itself a flailing industry. Valassis, which reported $2.8 billion in revenue last year, has seen sales declines for the past two quarters because of tighter marketing budgets. At the same time, more customers are moving dollars from print to online advertising and looking at new channels such as mobile devices. To keep up with that pace of change, and cut its costs, Valassis decided to consolidate its data center infrastructure, using virtualization to support more servers in less space and taking back ownership of systems that had been outsourced to an IBM data center. That project involved expanding the company's main data center near its Livonia, Mich., headquarters, bringing in so much new equipment that delivery-truck traffic jams formed in the parking lot some nights. By consolidating, Valassis cut $5 million a year from its IT infrastructure spending and $35 million from IT consulting fees related to a fragmented, partly outsourced data center, says CIO John Lieblang. While Valassis brought jobs back in-house in the process, that's rarely the case with infrastructure innovation. Usually, at least part of the cost savings comes from laying off IT staff, due to increased automation or efficiency. Bob Beauchamp, CEO of BMC Software, which makes IT management software, says that in a recent big deal BMC clinched, the customer's CFO wouldn't sign off until IT management provided him, in writing, with the specific positions that no longer would be needed as a result of the automation the software provided. Those people were laid off or moved to more revenue-focused or new development IT positions, Beauchamp says. Time Is Right To Consolidate Manufacturing and technology company Emerson recently completed a major data center consolidation, a departure from when decentralized IT was considered necessary. "A lot of decisions were being made at individual units, so that we could react to changing market conditions with a lot more speed," says CIO Steve Hassell. But that structure produced too much waste. After multiple acquisitions, the company had 135 data centers across the country. So Emerson decided to consolidate into a massive data center in St. Louis, and as part of the process did something a bit different--had the IT staff meet with the building architects. One of the most innovative decisions to come out of those meetings was to have the building's heat exchangers sit on the roof directly over the air conditioning units they support. Architects were initially wary of the idea, since that's not how it's done; typically, heat exchangers are on the ground behind a building. But putting them on the roof would let Emerson take a modular approach to expansion, adding additional cooling space only as needed as it gradually shuts down and incorporates other centers into the main one. The modular approach isn't practical using equipment on the ground, since it would require digging up the ground to lay pipe for each installation. The standard approach would've meant buying all the exchangers up front and laying miles of copper pipe underground for them, knowing they wouldn't come online for years as data centers were wound down. The architect team acquiesced. The data center opened this summer and will initially use only one-third of its capacity, but as a result of the roof idea, it isn't using any more air conditioning than it needs, Hassell says. What's more, putting the heat exchangers so close to the AC units has saved Emerson 2.5 miles in copper piping that would've gone into the ground. "Not only are we saving money in a downturn, but more importantly, we've provided a strategic platform to consolidate and grow as the economy turns around," Hassell says. The Economics Of IT In some instances, a lower-cost infrastructure already is fueling growth-oriented innovations. Investment bank Cowen is using Linux-based Ingres databases as it expands its group of program traders and develops a portal for the group to share analytics with customers. Program traders perform complex analytics to help fund managers make decisions about where to invest money, work that involves very large data sets. Ingres is mature enough to handle that workload, but cheap enough to help Cowen grow the program, says CIO Dan Flax. Nationwide Insurance's William Miller say he's seeing a lot more interest these days from business execs in IT infrastructure issues, which makes it easier to get these projects going. "It's reminiscent of the Y2K scramble, where a lot of tech-ignorant people got tech savvy really quick," says Miller, associate VP and controller for IT at the insurer. "My hope is that we don't lose that when margins fatten up." Now more than ever, it's important for IT and business to have a tough dialogue about infrastructure investments, Miller says. He cites the example of a business leader that demands a highly redundant server and storage system so the application won't go down, but the application isn't critical enough to merit the cost. "What you really want to do is expose the business to the economics of IT in a way that makes sense and is understandable to them," Miller says. The company uses software from Digital Fuel to track the cost and use of IT by individual business units. "It's not a cost pressure. It's a value pressure," says Miller, "You spend more if there is value in the investment. Companies that don't understand that relationship flat out aren't making it in any industry they're in." No doubt business is slow. Often brutally so. But companies can't let their IT infrastructure fall behind their competitors'. "The bottom line is budgets may be constrained, but the demands on technology are growing," says JPMorgan Chase's Chiarello. "It's not about spending more. It's about spending smarter." That means recognizing the unique opportunities that come with a down economy, and most importantly, understanding that the infrastructure isn't just about keeping the lights on. It's about tuning the engine that makes innovation possible.