Share with your friends


Analytics Magazine

Forum: Oil & gas producers need to tame the gusher … of data

July/August 2013

Warren WilsonBy Warren Wilson

Throughout the century-and-a-half since the dawn of the commercial petroleum industry, oilmen have always hoped for the gusher – the big find that would spew enough oil to make them rich. There have always been far more “dry holes” than gushers, however, and the proportion only gets worse as oil and gas become harder to find and more difficult to produce. One thing the industry has in abundance today, however, is data. And just like some of the biggest oil discoveries, the data gusher offers huge promise if it can be tamed. However, the exploration and production (E&P) industry, and the IT companies that support it, have a lot of work ahead to derive maximum value from their growing troves of data.

The first oil wells didn’t rely on data at all. They were sited where oil seeped from the ground naturally, and the challenge – not simple, but not data-dependent – was figuring out how to dig or drill down to the source.

Data grew more important as the industry grew, reserves became harder to exploit and drilling technology evolved. Drillers began keeping paper records of what occurred during each work shift or “tour.” They tracked basic metrics such as the number of feet drilled per hour or day, obstacles encountered, injuries sustained.

Over the years recordkeeping has become steadily more thorough and sophisticated. Today the E&P industry has vastly better tools for every aspect of oilfield operations including: three-dimensional maps of subsurface geological structures and hydrocarbon reservoirs; graphs or “logs” of the wells’ downhole conditions (temperature, pressure, porosity, permeability, etc.); and records of injuries and environmental incidents. New equipment is often fitted with sensors that produce steady streams of data about temperature, vibration and other parameters that indicate whether the asset is operating as it should, or whether it needs service, repair or replacement.

Yet E&P companies still find themselves in the same position as their predecessors and their counterparts in other industries: They are awash in data, but they don’t have all the insight they need. For example, they have copious real-time data from individual wells, but they do not have a good handle on the dynamics of complex unconventional reservoirs. Similarly, they have access to vastly better analytical tools than ever before, but few companies can claim to have fully optimized field operations, production or asset management.

Analytics Tools Must Evolve

E&P companies’ drilling programs have always relied primarily on historical data that describe a given region and its history. Extrapolation and interpolation suggest where to drill next – if you drill between two producing wells, your odds of success are relatively good. Outside known reservoir boundaries, the odds fall off dramatically; so-called “wildcat” wells face a much greater risk of producing no return on many millions of dollars invested. That has always been the biggest gamble in oil exploration – even when armed with the best historical, descriptive data available, you still have to spend large sums up front to drill the well before finding out if your data is any good.

Today the industry’s data and analytics needs are changing, and so are the tools at its disposal. Having found and developed most of the world’s “easy” oil and gas reserves, E&P companies are venturing into even more remote locations and extracting hydrocarbons from unconventional sources. They are tapping into shale rock so impermeable that it won’t give up its gas and oil without first being “fractured” with water, sand and chemicals injected under high pressure. They are tapping sand deposits that contain oil so viscous that it won’t flow without first being diluted with solvents or softened by steam.

These unconventional sources pose unique challenges that require greater precision and real-time analytics – for example, to keep drill bits positioned precisely where they need to be within the shale “pay zone” and to control the placement, composition and pressure of fracking fluids to yield optimal results.

At the same time, E&P companies increasingly need predictive analytics tools, for a variety of purposes. They need to better understand how current production methods will affect long-term yield. They need more accurate predictions of asset behavior to improve continuity in drilling and production while minimizing the costs of spare equipment or service crews. Software vendors are increasingly offering analytics tools that address such needs.

But prediction is just one step forward. The next is so-called “prescriptive” analytics that go beyond merely predicting future behavior, to recommend the best course of action to achieve a desired result. Such capabilities are quite rare today and at a very early stage of development. But they promise to bring new levels of performance, not just in E&P but in many other industries as well, because they track the results of their recommendations and feed those results back into the prescriptive algorithms to produce (in theory, at least) better and better recommendations over time.

Complex Needs Drive Demand for New Solutions

Despite all the potential benefits of advanced analytics in oilfield operations, their adoption is still at a very early stage. That is true of most industries, simply because the technologies themselves are new. Software vendors today are evangelizing their predictive capabilities; only a tiny number yet offer “prescriptive” analytics. Indeed, the term itself is not yet widely known.

An additional factor in IT adoption in E&P is the industry’s insularity. It is a world unto itself, an industry that operates largely outside of everyday view and speaks a specialized language that few outsiders understand. It is an industry driven by geologists and engineers who tend to regard IT as a mere support tool for existing operations.

The idea that IT, and particularly analytical software, can provide strategic and competitive advantage is not widely held. But until it is, E&P companies will not find or produce as much oil and gas as they could, nor will they manage their operations as efficiently and safely as they could. E&P companies owe it to themselves, and to their shareholders, to broaden their traditional view of IT and consider the strategic advantage it can provide. IT vendors, for their part, must state their value propositions in terms that E&P engineers can understand. They must explain, in plain language, how IT can help E&P companies find and extract more oil.

Warren Wilson ( leads Ovum’s Energy team, focusing primarily on IT for upstream Oil & Gas. He has been an analyst for 14 years. He joined Ovum in 2006 when Ovum acquired his former employer, Summit Strategies. At Summit his primary area of responsibility was mobile business applications. On joining Ovum, his research focus shifted to business management applications such as enterprise resource planning, supply chain management, and analytics.

Before becoming an IT analyst, Wilson had been a reporter and editor for U.S. newspapers including the Casper (Wyoming) Star-Tribune, where he covered oil & gas and other energy industries. He majored in geology at Carleton College in Northfield, Minn., and later worked in the oilfield as a roughneck and well logger.

business analytics news and articles



Related Posts

  • 81
    November/December 2014 The marriage of two “natural resources” – hydrocarbons and data – will transform unconventional oil development. By Atanu Basu (right), Daniel Mohan and Marc Marshall Known-knowns, known-unknowns and unknown-unknowns. Donald Rumsfeld’s notable turn of phrase is an apt characterization of where we are with unconventional oil development today.…
    Tags: data, oil, analytics, prescriptive, wells, gas
  • 62
    March/April 2014 By Atanu Basu The United States is re-emerging as an energy superpower. According to the International Energy Agency, by 2016 the U.S. will surpass Saudi Arabia and become the world’s largest oil producer. The domestic energy industry’s recent rise is the result of lower demand through energy efficiency…
    Tags: oil, analytics, data, production, gas, prescriptive
  • 58
    As the Big Data Analytics space continues to evolve, one of the breakthrough technologies that businesses will be talking about in the coming years is prescriptive analytics. The promise of prescriptive analytics is certainly alluring: it enables decision-makers to not only look into the future of their mission critical processes…
    Tags: analytics, prescriptive, data
  • 53
    The information explosion has led organizations to leverage data to improve the overall decision-making process. Organizations are looking to deploy data-driven strategies across their business processes and functions, such as marketing, risk, supply chain and finance. It’s critical now more than ever to have the right people, processes, methodologies, platforms…
    Tags: analytics, prescriptive
  • 49
    November/December 2012 The advent of the “digital oil field” helps produce cost-effective energy while addressing safety and environmental concerns. Finding and producing hydrocarbons is technically challenging and economically risky. By Adam Farris Everyone needs it, few know how we get it, and many feel compelled to slow down efforts to…
    Tags: data, oil, gas, industry


Report: One in five cloud-based user accounts may be fake

According to the Q2 2018 DataVisor Fraud Index Report, more than one in five user accounts set up through cloud service providers may be fraudulent. The report, based on information gathered between April and June, analyzes 1.1 billion active user accounts, 1.5 million email domains, 231,000 device types and 562 cloud hosting providers and data centers, among other indicators. Read more →

When managers respond to online critics, more negative reviews ensue

A new study in the INFORMS journal Marketing Science found that when managers respond to online reviews it’s possible that those responses could actually stimulate additional reviewing activity and an increased number of negative reviews. The study, “Channels of Impact: User Reviews When Quality is Dynamic and Managers Respond,” is authored by Judith Chevalier of the Yale School of Management and NBER, Yaniv Dover of the Hebrew University of Jerusalem and Dina Mayzlin of the Marshal School of Business at the University of Southern California. Read more →

IE student designs software to optimize snow removal at Penn State

It is well known among the State College and Penn State communities that it takes a lot for university officials to shut the campus down after a major snowfall. In fact, since 2010, the University Park campus has been shut down just three full days due to snowfall. Much to the chagrin of students – and faculty and staff – the snow day at Penn State may just have become even more elusive, thanks to software developed by recent industrial engineering graduate Achal Goel. Read more →



Winter Simulation Conference
Dec. 9-12, 2018, Gothenburg, Sweden

INFORMS Computing Society Conference
Jan. 6-8, 2019; Knoxville, Tenn.

INFORMS Conference on Business Analytics & Operations Research
April 14-16, 2019; Austin, Texas

INFORMS International Conference
June 9-12, 2019; Cancun, Mexico

INFORMS Marketing Science Conference
June 20-22; Rome, Italy

INFORMS Applied Probability Conference
July 2-4, 2019; Brisbane, Australia

INFORMS Healthcare Conference
July 27-29, 2019; Boston, Mass.

2019 INFORMS Annual Meeting
Oct. 20-23, 2019; Seattle, Wash.

Winter Simulation Conference
Dec. 8-11, 2019: National Harbor, Md.


Applied AI & Machine Learning | Comprehensive
Dec. 3, 2018 (live online)

Advancing the Analytics-Driven Organization
Jan. 28–31, 2019, 1 p.m.– 5 p.m. (live online)


CAP® Exam computer-based testing sites are available in 700 locations worldwide. Take the exam close to home and on your schedule:

For more information, go to