Share with your friends


Analytics Magazine

Messy analytics? It’s OK. We’re housewives!

September/October 2013

Information systems and technology are messy, which makes applying analytics an arduous task.

Information systems and technology are messy, which makes applying analytics an arduous task

Gary Cokins

By Gary Cokins

Who is working with perfect systems? That is like asking if your house or apartment is always clean and tidy to be presentable to guests. In your organization, who cleans up messes? Who fixes things?

I am writing on this topic of messes based on a cafeteria-style breakfast I had in a hotel in Estonia where I was presenting a seminar. It was the peak morning rush, and all the tables were filled with diners. A couple that was seated next to me had just departed. Two women who I’d estimate were in their late 40s eyeballed the two vacant chairs. One made a silent inquiry, “Are these seats available?”

I promptly answered, “Yes, but it’s a mess” and pointed to the soiled dishes, coffee cups and silverware. One of the ladies immediately replied, “It’s OK. We’re housewives.” They then picked up the soiled dishes and carried them to a nearby cart.

After she replied, I had to laugh out loud. Her reply communicated so many things to me. It meant, “We are accustomed to cleaning up messes left behind by others.” It also meant, “Someone has to do it,” especially when it is urgent. It also meant, “If you have been cleaning up most of your life (and I suggest the reader replace “life” with “career”), then you are resigned to the reality that there will always be a mess or problem that will need to be addressed, cleaned or fixed.”

What Type of Mess are you Cleaning?

In information systems and technology (IS/IT) the messes are on three levels: technical, analytical and managerial. I will describe them soon, but first let’s note that there are two types of IS/IT cleaning and maintenance staff.

1. Hardware and infrastructure professionals “cleaners.” Candidly, I am not knowledgeable enough to understand what they do and what their jobs entail. I suspect it is complicated. To me it is like they are down in the boiler room of an ocean sailing ship. To oversimplify their job, it is to keep the lights on for the users.

2. Analyst “cleaners.” Their job is to convert data into information with the purpose to provide insight and foresight for better decisions and actions.
Both types are important. They serve different purposes.

When you read my classification of “mess” levels that follow, which of the three levels do you believe is the most damaging to an organization’s performance and success? Here they are:

• The technical mess. Technical messes involve disparate data sources, a patchwork of purchased hardware and software with predictable incompatibility problems and computing/storage capacity issues. Down in the IS/IT “boiler room” there are lots of hammers and wrenches to fix things.

The analytical mess. In contrast to the technical mess, which is typically complicated to deal with, the analytical mess is complex. (To learn the difference between complicated and complex in this context, see [1].) In summary, complicated systems have many moving parts, like a wristwatch with gears; but they operate in patterned ways. In contrast, complex systems have patterned ways, but the interactions – think variables – are continually changing. In the former, one can usually predict outcomes. The math may be easy with linear relationships. In the latter, such as air traffic control, weather and aircraft maintenance delays cause changes in the constant interactions with numerous variables.

Analyst messes are partly caused by technical messes. An example is defective and/or incomplete source data – the dirty data mess. But even if the analyst has the luxury of perfect input data quality (like a germ-free hospital surgery room or semiconductor chip clean room environment), they will still encounter a problem framing the mess. A key aspect of an analyst’s job is to correctly frame the problem they are trying to solve. Solutions to problems do not always require nails, so an analyst needs more in their tool belt than a hammer.

The good news for analysts is that high-performance analytics and visualization software is now leveraging massively high-speed computing and storage technology. The combination is like a powerful cleanser. With this advance in technology, instead of the analyst framing a problem with a carefully thoughtful hypothesis to test variables that used to require hours of computer time, today it can be run in seconds. This means that analysts can more quickly test and learn. Better yet, they can quickly test, fail and learn. Failing to validate a hypothesis no longer has the adverse consequences it once had in terms of causing delays. Analysts can now fail so quickly that no one sees the brief mess they made.

The managerial mess. Senior management can create a mess that is more difficult to clean compared to a technical or analytical mess. This is because the mess they create comes from their minds and attitudes.

The managerial mess has two broad categories: power and incompetence.

The mess caused by power typically involves such a high reliance on intuition, gut feel and past experience that managers believe they can get by without fact-based information and deep analysis. Confirmation bias further muddies the floor when executives nudge the analysts to twist the findings to satisfy the executive’s pre-conceived notion of what is the correct answer.

Messes caused by incompetence typically involve the inability to see or understand what the analyst’s findings imply for decisions and actions. For example, when a marketing analyst for a telecommunications company examines more than a thousand variables to determine what types of optimal deals might best be offered to maximize profits from differentiated types of customers, the findings may be counter-intuitive – yet valid. That is because analytics deals with complexity, not just complications.

Both managerial mess categories can be addressed, but it requires fortitude by employees to manage their managers; that is, to convince managers of the follies of their management style.

Cleaning Solutions for Messes

Technical messes can be cleaned with good, advanced capital investment planning for purchasing (or now with SaaS, renting) the correct hardware and software. Also educate, educate, educate; and train, train, train.

Analytical messes can be cleaned with good data governance practices (e.g., extraction, transform and load) and with high-performing analytics software with visualization to accelerate the analyst’s experimentation and investigation.

Managerial messes are more problematic. The stains in the carpet are deep.

After the long, cold winter, we all attempt to do spring cleaning. I submit that following the global economic collapse of 2008 many organizations are on the road to recovery. They experienced their winter of the economic cycle. It is time to clean up messes.

We need to behave like the two breakfast diners I met who said, “It’s OK. We’re housewives.” Get on with the task. Get your IT house in order. The popularity of analytics, operations research techniques and big data is quickly growing. Having messes will slow down your progress to leverage and deploy analytics.

Gary Cokins ( is president of Analytics-Based Performance Management LLC in Cary, N.C. He is a member of INFORMS.

business analytics news and articles


Using machine learning and optimization to improve refugee integration

Andrew C. Trapp, a professor at the Foisie Business School at Worcester Polytechnic Institute (WPI), received a $320,000 National Science Foundation (NSF) grant to develop a computational tool to help humanitarian aid organizations significantly improve refugees’ chances of successfully resettling and integrating into a new country. Built upon ongoing work with an international team of computer scientists and economists, the tool integrates machine learning and optimization algorithms, along with complex computation of data, to match refugees to communities where they will find appropriate resources, including employment opportunities. Read more →

Gartner releases Healthcare Supply Chain Top 25 rankings

Gartner, Inc. has released its 10th annual Healthcare Supply Chain Top 25 ranking. The rankings recognize organizations across the healthcare value chain that demonstrate leadership in improving human life at sustainable costs. “Healthcare supply chains today face a multitude of challenges: increasing cost pressures and patient expectations, as well as the need to keep up with rapid technology advancement, to name just a few,” says Stephen Meyer, senior director at Gartner. Read more →

Meet CIMON, the first AI-powered astronaut assistant

CIMON, the world’s first artificial intelligence-enabled astronaut assistant, made its debut aboard the International Space Station. The ISS’s newest crew member, developed and built in Germany, was called into action on Nov. 15 with the command, “Wake up, CIMON!,” by German ESA astronaut Alexander Gerst, who has been living and working on the ISS since June 8. Read more →



INFORMS Computing Society Conference
Jan. 6-8, 2019; Knoxville, Tenn.

INFORMS Conference on Business Analytics & Operations Research
April 14-16, 2019; Austin, Texas

INFORMS International Conference
June 9-12, 2019; Cancun, Mexico

INFORMS Marketing Science Conference
June 20-22; Rome, Italy

INFORMS Applied Probability Conference
July 2-4, 2019; Brisbane, Australia

INFORMS Healthcare Conference
July 27-29, 2019; Boston, Mass.

2019 INFORMS Annual Meeting
Oct. 20-23, 2019; Seattle, Wash.

Winter Simulation Conference
Dec. 8-11, 2019: National Harbor, Md.


Advancing the Analytics-Driven Organization
Jan. 28–31, 2019, 1 p.m.– 5 p.m. (live online)


CAP® Exam computer-based testing sites are available in 700 locations worldwide. Take the exam close to home and on your schedule:

For more information, go to