What is Big Data?

Big Data is a wonderful term, but what does it mean? Many IT professionals will say that they have been providing systems which capture, analyse and present data -- often in very "Big" quantities -- for years. What's new? I think the most useful definition of a Big Data system is a very pragmatic one: you need a Big Data system to capture, analyse and present information when traditional Relational Database centered systems cannot meet your Users' requirements. The reason Big Data is such a current topic right now is because so many businesses are running up against these limits and being forced to look beyond them. Why might this be? Here are the three key drivers, in my opinion. We will look at each in much more detail over the course of this blog:

1. First and foremost is time. Time comes in two flavours: Latency -- how long it takes to get your data -- and what I'll call "Freshness" -- how up-to-date it is when you get it. If you cannot meet your Users' time expectations with traditional technologies, then you are into the realm of Big Data. These expectations are becoming more demanding, and data volume, which  always costs time, is growing by orders of magnitude. We will talk a lot more about time in this blog.
2. Second is flexibility. The need to be flexible and adaptable can overload a traditional approach to data systems in three ways: Variety of Sources, Variety of Structures and Future Uncertainty. A common requirement for Big Data systems is that they collect their data from an enormous variety of sources. This data comes organised in a similarly large variety of structures or "schemas". While it is possible to come up with ingenious ways to get data from many sources into relational databases, these databases do not take kindly to data which does not conform to a predefined structure, or schema. Unfortunately, Big Data systems are often required to accept data coming in a huge variety of structures, and often have to deal with new, previously unanticipated, data structures and analysis requirements as time goes on.
3. Third is complexity. What we are talking about here is the ability to model complex real-world behaviour and perform complex analyses, often in real time. Not something talked about a lot in the world of Big Data at the moment, but I believe will become more and more important as the applications of these systems become more sophisticated, and again, not something relational database technology, with its focus on selection of lists and relatively simple aggregations -- average, max, count, etc -- is ideally suited for.

So in summary, Big Data systems are by definition different in kind from 'traditional', relational database centric systems, and are driven to be so by User requirements which beyond certain limits of volume, structure and sophistication, are not deliverable by this 'traditional' technology. So if you are an IT professional, and your User or Customer is asking for something not achievable with your existing tool-set, it is not because they are wrong, it's because they are asking for Big Data!

Big Data

I have been quiet on the Fraysen blog for some time now, but very busy in the 'real world!' While we have been working away with Clients and on new product development, the market for the software Fraysen develops has aquired a name: "Big Data." I have spent some time thinking about how I can best contribute to the Big Data discussion. It is a very broad term, spanning a value chain of systems which can overwhelm and confuse, so my first goal will to try to bring some clarity and simplicity to the conversation, give you some reference points, and a context to understand it in. I hope you find it useful.

OEE: how to approach root cause analysis?

An interesting question was posted on the LinkedIn MES - Manufacturing Execution Systems group recently by Elisa Rocca of Siemens: "OEE: how to approach root cause analysis?" She goes on to say that "I was wondering how to best execute the root cause analysis for a filling & packaging line." and later in the discussion that:

"I would like to decouple the RCA investigation from levels 1 and 2 [PLCs and SCADA systems directly controlling the machines - FG], conducting the entire analysis into MES.

...  I'm wondering whether you see the RCA as an "online" analysis performed during the acquisition of OEE data, or as an "offline" analysis done on historical data using reporting tools"


My Response:

The challenges with root cause analysis of any live process metric, including OEE, are:

1. A priori you do not know what data you are going to require to solve the problems that exist, since you do not know what these problems are!

2. You will have a range of users for this RCA, from Operators and Supervisors, who's needs will be near real time, to Engineers, who will perform their analysis mainly historically.

3. You will probably need to combine 'hard' data from your equipment with judgement / big picture input from humans in the process.

4. You need to ensure that the effort required to perform the RCA is minimized: the more work required, the less likely it is to get done, and the greater the opportunity for confusion and arguments!

What does this suggest?

You are right to decouple your RCA from levels 1 & 2: they are great for machine management, but are too inflexible, machine focused and simplistic (analytically) to provide the solution you need. Similarly, ERP (in a manufacturing context) and MES systems are focused on dispatching, routing and recording production, rather than analysing the performance of the process.

Your requirements are the motivation for Enterprise Manufacturing Intelligence systems. I would suggest looking for EMI systems and rating them by the following attributes:

1. Flexibility -- could a Process Engineer configure their own analytics?

2. Scalability -- you will rapidly scale to large numbers of data sources and volumes of data. Is the system architected to cope with this?

3. Analytical Power -- you are going to need to analyse your process along many axes: machine, product, batch, shift, material, etc, and to perform sophisticated analyses to correlate causes and effects. Does the system provide the power you need in its tool set?

4. Ability to combine data from many sources, including friendly interfaces for human operators. Is there an easy way to capture human input and combine it with machine data?

5. Automation -- the system should do 80% of the analytical work automatically, especially the 'drudge' number crunching that does not require human judgement, otherwise this will not get done at all! How is this handled, and how long does the user have to wait for the results? Again, if the answer is >> a minute, the User is unlikely to wait, no matter how useful your data!

See here for the full discussion.

Impressive Opportunity from a Small Experiment

Just published an analysis on the Fraysen Systems website today called "Material Flow Co-ordination Case Study" which gives the results -- with their agreement, of course -- of a quick opportunity analysis we did for one of our Clients. What is striking is:

1. The amount of information that we were able to collect from a simple experiement;
2. The size of the improvement opportunity we uncovered together.

There are huge cost reduction opportunities in many businesses today. Sometimes they are 'traditional' OEE improvement projects, sometimes they are a little bit different, in this case just making sure all the parts arrive at the same time. The other nice thing is that this opportunity doesn't even require a plant wide solution. A point application of the right Information Technology can make a world of difference!

There is a Gap between Operational Excellence and Business Reality

Why do Managers not rush to put into practice every recommendation by their Lean Consultants, or to install Manufacturing and Business Intelligence systems? There is plenty of data to suggest that in general these methods and systems produce outstanding returns, and this is backed up by detailed analyses of selected leading companies.

I think there are just too many "IF"'s between the proposition and the return. In essence, when a Lean Consultant presents their report, they are saying "if you invest management time in changing your organisation's behaviour, and if you get your people to behave in this way, and if the consistently apply these methods even if it appears more expedient in a crisis to go back to the old ways, then you will reap the benefit of my advice." Similarly vendors of Operational and Business Intelligence tools are saying "if we pick the right KPIs (Key Performance Indicators) to monitor, and if you make monitoring these KPIs part of your people's management process, and if they respond quickly and appropriately when they are alerted to an issue, and if they carry the action through to closure, then our system will deliver a return on your investment."

Think about it: in a world where at least on the face of it a new machine will increase your throughput because it's advertised cycle time is 20% lower, or you can save millions in labour costs by moving to China because they earn 1/10 of people in the West, it is asking a lot of your Client to make such a complicated and contingent cased to their Board / Senior Management and then ask them to invest money in it. Should we just give up? No - we just need to make a better case, and I believe that by taking responsibility for the Client's problem, rather than limiting ourselves to being just a part of the solution, we can!

The Operations Management Dilemma

Having spent a large part of my career in developing and selling Operational Intelligence Systems, I am one of many businesspeople around the world -- Manufacturing Systems vendors, Business Intelligence providers, Lean Manufacturing consultants and Six Sigma Black Belts who make their money persuading corporations to focus senior management attention on how they operate, and to spend money on ways to improve it. After all, a company's operations are what deliver value to its Customers and Owners, so why wouldn't they want to invest effort and resources in making them better. Right?

A presentation by Eoin O'Driscoll, (chair of the Irish Government's Enterprise Strategy Review Group) at the 2009 Midwest Entrepreneur Showcase gave me pause. In discussing how to create value for Customers, he referred to Peter Drucker's assertion that "Because its purpose is to create a customer, the business enterprise has two—and only these two—basic functions: marketing and innovation. Marketing and innovation produce results; all the rest are costs" (my italics). Therefore we in the "Operational Excellence" industry are trying to persuade corporations to spend scarce management time thinking about things that they do not consider core to their success, and then follow that up by asking them to invest even more money into an area of the business they already consider to be primarily a cost. What’s wrong with this picture?

While the issue is not quite as black and white as I describe above -- otherwise there would be no manufacturing software companies or Lean Six Sigma consultants at all -- it does jib to a startling degree with what we see in the market. In 2004, average western manufacturing efficiency was quoted as 45%, with world class companies at 85%. The figures have not changed that much since; the opportunity has been obvious for decades -- see the US vs. Japanese auto industry saga -- and yet many companies still at best reluctantly invest leadership time and effort into Operational Excellence as it's currently sold. They don't want what the industry is trying to sell them. Could they be right?

High Velocity Business

I had the pleasure to read Steven J. Spear's "Chasing the Rabbit" during my Summer vacation, in which he discusses the unique attributes of industry leading companies like Toyota and Alcoa. What Mr. Spears wanted to understand is why these companies have maintained and even extended their competitive advantage over their peers, even as these peers have learned from these leaders and themselves improved. Interestingly, what he singles out as unique about these companies is not that they apply methodologies such as Lean and Six Sigma, although they do, and very effectively, but the fact that they apply these methods in the context of a business culture which is focused on learning and improvement, from the 'highest' to the 'lowest', in everything they do.

What these companies do is simply to recognise that their businesses are complex systems which it is impossible to design perfectly from the outset. So instead of pretending that the system is perfect, and making do as best they can, they accept that there are going to be opportunities for improvement, and invest in their ability to spot, understand, address and learn from these opportunities. As a result, whereas their competitors come to work on a Tuesday at a similar level of performance as on Monday, and expecting to have to overcome most of the same problems as faced them on Monday, the High Velocity Business is coming to work having solved at least some of Mondays problems for ever, and therefore is starting the day at a higher level. These improvements quickly accumulate over time, propelling the High Velocity Business forward even as competitors repeatedly attempt to catch up.