Migration from analog to digital products and services requires handling and extracting insights from an unprecedented volume of data.
Leading companies worldwide are redesigning their infrastructure and organization to adapt their business to the digitization wave. Indeed, the migration from analog to digital products and services requires handling and extracting insights from an unprecedented volume of data.
In the last couple of years, the advent of open-source technologies fueled big data initiatives with the intent to materialize new business models. Although, there is consensus around the potential business opportunity behind this megatrend, there is not on how to seize value from it. As a software vendor, we believe that technology is a means and not an end.
In the next pages, we will highlight a set of best practices (questions to ask yourself, tips and pieces of advice) to avoid being trapped in the ever-lasting big data project that fails to generate any revenue. Our experience comes from the telecoms industry – one of the most data intensive environments – but the addressed concepts can be transposed to other verticals.
The current digitization wave provides a new space for business development. In fact, the phenomenon materializes differently whether traditional or digital native players are considered:
For traditional players, it allows them to:
- Increase functional coverage
- Build multichannel capability
- Enrich the customer experience
For digital native players, it offers capabilities to develop new business models:
- Based on a leaner cost structure
- Natively built around enhanced customer knowledge
Cards, sensors and smart devices, enable the collection and transmission (through GSM, ATM, RFID, etc.) of data records from daily events. So far, the technical capabilities available have not allowed processing such a variety and volume of data.
The current evolution in ICT allows businesses to store and analyze data to extract valuable insights. These have the primary objective of enhancing a company’s activities in order to provide better products and services.
There is no wonder why mobile operators are all interested in Big Data: the huge amount of data they collect from networks and IT opens the door to fascinating usages.
Big data software deployments drive both revenue generation and cost reduction. On the revenue side, they can lead to internal or external monetization. And on the cost-saving side, they impact marketing, network, fraud management and customer experience.
All these fascinating use cases raise many questions from a technical, financial and organizational standpoint. Let’s go through some of the common questions and challenges operators are facing when starting a Big Data project.
1. The Challenge of Fragmented Insights
The first challenge one can think of is technical: collecting such a wide variety of data from so many different sources can look terribly complex. Have you ever considered mixing very different siloes of data such as customer centric databases, transaction detail records, network information, and data from external website, partners, etc.?
As a matter of fact, our experience shows this technical question is not the hardest one to solve. There are many ways to ingest massive streams of data into a single data entry. The great challenge is elsewhere and deals with more trivial questions.
- Considering the data, most of them are already collected by the telecom operator. However, these collection points were designed for a specific operational purpose and not meant to be loaded with heavy queries. So should operators rely on their existing systems and bear with slow response time? Or should they rather shortcut them and build parallel collection directly from the sources?
- Another non-technical issue would be that many information is made redundant in an operator’s IT. For example, the customers’ ARPU (Average Revenue Per User) are calculated in many different systems with different rules, according to the indicator’s end use. Therefore, it is not so much about finding where the data is, rather than choosing which of the different sources will be taken as a reference. Whatever the choice will be, operators need to bear in mind that it will never be a 100% consistent with other KPIs already in place across the company.
2. Build or Buy?
It is possible to successfully build big data infrastructures from scratch. Companies like Google or Yahoo for instance have chosen to use Hadoop. However, even though Hadoop has matured over the past years, our experience is that successful Hadoop implementations put into full production using do-it-yourself infrastructure components always require more time and effort than initially expected.
Of course, it is not expensive (low infrastructure cost) and it can be downloaded for free with easy access. But the other side of the coin is that it requires time, resources, and expertise to build up a do-it-yourself Hadoop infrastructure big data project. The question of skills is all the more important as industry reports shortages of skilled staff in all links of the chain (IT, BI and analytics, database administration…).
In 2016, despite an apparent growing talent pool, companies have to offer bigger salaries in order to attract Hadoop skilled talents and have to go all-out to retain them. So here we are, demand for developers and infrastructure specialists with open source skills is continuing to grow, according to the 2016 Open Source Jobs Report, and organizations are still struggling to fill positions.
If building your own solution feels safer because you master your environment, this choice usually comes with negative impacts, as it involves a higher time-to-market and hidden costs that can counterbalance the benefits. This is such a difficult choice to make that some of our customers started with an internal project and ended up with off-the-shelf solutions. We believe this is one of the cleverest ways to go.
3. Why Big with Fast Data?
Most of the time, examples of Big Data refer to data lakes: storing Peta bytes of information on the fly, and, from time-to-time, running a powerful algorithm on these huge data bases to get a clear picture on segmentation, risks, scores, etc. This approach is adapted to studies that do not require an immediate trigger for action.
However, there is another approach consisting not only in storing but also in analyzing the data while it’s still flowing, to detect patterns. This Fast Data approach is particularly relevant when timing is of the essence, for example to detect fraud, to grant authorization, to notify a customer that he has reached a certain threshold of usage or a credit limit, etc.
These applications require real-time patterns detection but also the capacity to trigger a set of relevant actions on-the-fly. Different technologies can be used depending on the applications the company wishes to set up.
There are two ways in considering a Big Data project. Either start small and launch the project based on one or two profitable use cases, or consider it a must have for your company and build large storage capabilities, designed to cope with unknown future use cases.
Of course usually operators opt for a mix between these extremes, but it is interesting to examine the consequences of one choice or the other:
When operators opt for a use case approach, they are driven by a clear goal and usually their commitment to deliver is stronger. By definition, they are less challenged on the ROI question afterwards.
However, it can be challenging to find the relevant use case because the business units can have different priorities or because they wish to address their use case on a specific platform. Or simply because the business units do not share the same enthusiasm about the business opportunity than the Big Data project manager. In any case, our experience is that any business plan has limits and the future is never as it was planned to be. So our recommendation is to ensure both the technical solution and the organization will be compatible with many different usages. Far beyond what was initially planned. />
The other approach consists in storing everything and bet that new usages will emerge from this lake of raw information. Usually this approach provides larger degrees of freedom to try innovative use cases. And it can give a bit more time to freely explore the data before deciding which next use cases will be rolled out.
Experience has taught us that the lack of commitment on ROI can slow down the decision to launch the project or to make clear trade-off between multiple choices. The risk is to have fuzzy objectives that may change in time.
There is nothing wrong about this approach, but it demands strong buy-in from business lines and a clear governance model to make sure the project keeps on track.
5. Organization & Governance
What about the organization of a big data project? Should the project be carried out by the IT department or should it be led by a dedicated organization, under a new function like a Chief Data Officer, distinct from traditional IT? Of course there is no magical recipe for success.
In our experience, whenever IT department leads the project, they benefit from a natural legitimacy in terms of technical choices. They also have a larger flexibility on resources, because a Big Data project requires significantly less investment than a core IT system. They may also experience a lower pressure on ROI or new use cases delivery.
On the other hand, dedicating a special team to Big Data projects can be an interesting alternative, because it comes with a clear objective and is usually driven by a P&L. Mixing different competencies within the same organization can impulse an innovative mindset, with clear incentive to deliver results. The drawback is that a dedicated team may struggle to convince other departments – either IT or business lines- to use a specific platform rather than alternative solutions… So usually this approach is faster to deliver the first results, but can lose its momentum after a certain period of time.
To summarize, we have identified several success factors listed hereafter:
- Break the siloes & choose the right way to access the right source of data
- Check if your use cases are based on pure analytics or if they require real-time responsiveness
- You can have a use case approach but make sure your technical choices can address a larger variety of applications
- Give the lead to IT or a dedicated team, but in any case make sure governance is clear and business units buy into the project
Overall, be nimble. Stay flexible, in order to adapt and learn as quickly as technologies evolve. If a myriad of opportunities are available, it is important to keep in mind that you cannot explore all at the same time. Technologies are a means to unlock, develop and scale business. Knowing where you are now and want to go, while being down to earth is the path to success.