Gartner BI & Analytics Summit London – Day 1

What a great day, very exciting and lots learnt from industry experts, with very little bias towards a vendor or specific technology. This post will form the first of two which is my attempt to mind dump some of the key takeaways I have picked up and found useful here at the conference.

BICC – Business Intelligence Competency Centre

Is dead… long live the ACE – Analytics Community of Excellence. In the keynote Neil Chandler suggested four things wrong with the BICC, business, intelligence, competency and centre! Although it is still the driver behind successful, versus non-successful, BI programs it is not encompassing enough of the modern world of BI and Analytics. Key benefits of this approach is an attempt to drive BI programs from business outcomes but mostly it fails and still becomes an efficiency drive, not linked to delivering actual business value. It also does not encompass a new wave of change in the business, self-service, which is near impossible to centrally manage. Finally it has not been driven around the new future of analytical applications which are focussed around algorithms and a scientific approach to running of businesses, and our lives!

Whilst there is a lot in words the evolution from BICC to ACE is not just about words it is about evolving and improving a good concept and bringing it up-to-date and inline with business needs. Analytics now seems to embrace BI and gives us a larger maturity scale for businesses in our ever changing world. Community takes away the need to centrally control and helps with the, already in-place, self-service. Finally excellence is about striving towards something we perceive as the ultimate goal, not just a list of competencies that are based around technology.

One thing that hasn’t changed but cannot be ignored is that you MUST FOCUS around BUSINESS OUTCOMES to achieve excellence in your BI or Analytics program.

Algortithms are KEY

An important key theme for the keynote centred around algorithms and their use in BI and Analytics as well as in day to day life. Guessing which classical piece of music was generated by a computer or Bach highlighted how compute power and science has progressed and there is a real feeling from Gartner that through the use of algorithms and the tools that support them we can automate, improve and gain valuable insight into our businesses. Algorithms are used all over your business today, take some time to document them and look to find tooling to support their automation and improvement. Citizen data science communities may spring up around this.

Other key notes:

• IoT algorithms are set to generate $15 billion by 2018
• Over half of organisations will be leveraging algorithms by 2018
• By 2020 50% of Analytics leaders will be able to link their programs to real business value.
• The best analytics leaders can formulate new questions as well as answer existing business ones. They also fail, learn and push the envelope; I would add they fail fast, time is gone for multi-year BI programs.
• Data Management and Data Integration tools are converging, but not as fast as you may imagine.
• BI and Analytics tools are also converging but it was noted NO one vendor can support all BI and Analytics needs and the buyers of each are currently different.
• Quadrant analysts highlight IBM’s Watson Analytics as a good example of an analytical application.
• Microsoft’s Power BI v1 (that horrible O365, SharePoint online linked tool) failed, but v2 has gained traction and is having a negative impact on Tableau’s performance.

Still a lot of learning to go, even on day 1 but I wanted to share this for those not fortunate enough to attend this amazing event!

 

Building BI Projects with Agility

Let me introduce you to an interesting story that was part of the foreword of the Mike Cohn book “Succeeding with Agile: Software Development with Scrum”. In the story, trappers would stop off at the Hudson Bay store to get all their supplies before setting off into the wilderness and make their fortune. However once they had done their shopping at the supply store they would only go a few miles and then set up a camp. The idea was that they would be better to identify what they had missed in that initial shop and only have to go back a few miles rather than being lost in the wilderness without an important piece of equipment. Although most trappers, like most consultants, feel their planning is brilliant the really good ones prepare in a way that they don’t end up frozen in the Rockies!

I like this analogy to projects but does it really resonate with BI? For me it certainly does. On an older BI project we had units of measure to handle in our solution. Applying a Kimball design pattern I asked the company are there really only 3 UOMs you want to handle? The reply was a confident yes with confirmation that they had not reported on more than three for the last 5 years. Great! So based on this confident feedback I built the UOMs into columns in the fact table(s) rather than applying a UOM dimension. I created MDX for time intelligence for each date hierarchy and for each UOM, for three UOMs this wasn’t a hardship.

However, a few months later the customer requested my time to work on an enhancement request to add another 5 UOMs. Unfortunately the nature of consultancy “time & materials” means the customer doesn’t want to spend time (therefore money) on redesigning the core design and including a UOM dimension. So I was left in MDX script hell, although not with the usual complexities of writing MDX but in copy paste and having the longest MDX script in the western world (there must be longer ones in China with 1000s of characters in their writing system!).

If we had been more agile and delivered an earlier version of this cube (as this was the deliverable) then perhaps the business would have seen how easy things now were with these UOMs and suggested they would like to slice the data by more. We then would have had time to refactor the design and implement our UOM dimension.

BUT not all projects fall easily into a completely agile process. For example, did the trappers go to the Hudson store and buy just a sleeping bag (forgive me if these weren’t available in 1890) and then head off to camp to check it worked. Then they bought a kettle on day two before on day three realising they needed some tea bags… I don’t think so. Yet many software developers who talk of pure Agile say that there is value in delivering a web page with a username text box, no password box and no logon button… I don’t agree and would argue that there is value in QA and UAT at this point, but NO value in that delivery to the business.

With BI projects there may be some value, depending on the business requirements, to deliver a product dimension and let the business see this; I certainly found this with retailers that could then identify missing attributes and start to design hierarchies they wanted or with customers that had a customer dimension for the first time. Of course there was value in applying a new business definition of a unique customer and then seeing what this list of customers was for the first time.

However the key to both of these examples is that the biggest business value for these customers was looking at sales orders over time for these products or for identifying how customer recency affected the actions of this new list of customers. To delivery the biggest value to the business we have to make sure we have the tent, sleeping bag, cooking stove and, at least, a kettle to boil water. Everything needed to do a basic camping trip in the Rockies (I am not a camper). For me this is the high level design of the BI project and I suggest to deliver any value to the business the high level design needs to include everything you know you need at the start of the project. On later camping trips it is likely you will have more experience and that the initial design or shopping trip will be more detailed and there will be less trips back to the store.

From that point on, continually delivering value, based on business priority/need, is achievable. Breaking down complex tasks into work items and grouping these into sprints works really well but not just for the sake of unrelated work items. It is much better that a sprint focusses around a group of items that, together, may allow delivery of something that is valuable to the business. To do this you must be interacting constantly with the business. In a recent project we had an excellent MI team who helpfully worked with us to group work items in a set of business headings all relating to a key area that would add value to the team and therefore the business. We could then focus sprints around these headings and, once deployed, the MI team could start to see value in the BI project.

But the key on a new project is that without that high level design and that first big effort of choosing what you need, building it and then heading out to the wilderness, you are simply going camping with just a sleeping bag and will make more trips to the supply store than are really needed. Get good requirements from the right people (not IT), build as comprehensive a design as your experience and the requirements allow and then deliver the first cut for business review as early as possible. To help with this I often try and not worry about control processes, partitioning strategies, incremental ETL loads etc. I move that to later stages of the build so we can focus purely on delivering that first cut to the business.

In summary I do not think that a single “pure” process has been required across all my projects. But being more agile by using best practices from across the various methodologies has helped me to be more and more consistent in delivering high business value solutions to my customers. And just like evolving your development processes helps improve delivery always look to learn from each project to help evolve your way of working, this for me is one of the best things being Agile teaches us.