What is a Big Data Centre of Excellence?

A Big Data Centre of Excellence (BDCoE) is an enterprise function that takes an organization from zero knowledge to having a fully functional practice of Big Data technologies and processes to deliver robust business results. A BDCoE is where the organization identifies new technologies, learns about new skills and develops appropriate processes that are subsequently deployed throughout the other business units of the organization.

A BDCoE is essential to accelerating Big Data adoption by the enterprise in a fast and structured manner. It reduces the implementation times drastically and therefore the time-to-market to deploy new data-driven products and services. More importantly, it ensures that the best practices and methodologies are shared through different teams in the organization. A BDCoE should be a live and evolving organizational function that expands and grows as the organization’s needs evolve.

A centralized BDCoE can be the foundation for establishing a data driven enterprise that values data as its strategic asset. The BDCoE can partner with the business to identify which projects should be prioritized and what data is of strategic importance. As such, it operates as the strategic counterpart of the business to translate current and actual business requirement into live and actionable Big Data projects. Big Data’s strategic importance is the value it represents for the business, but success with Big Data is not just about data. The people and the organization also play a vital role in that success.

How to set up a Big Data Centre of Excellence

An effective BDCoE consists of five major pillars that together form the structure for obtaining value from the centralized function.

Big Data Teams

The most important element, the quality of the Big Data analysts, Big Data scientists and Big Data engineers is paramount for creating success with Big Data. In the end, Big Data is knowledge domain, and that knowledge will come from the people. The Big Data professionals needs to be certified and experienced practitioners that have a track record of working with data.

Big Data Labs

Big Data labs refer to the working environment of the BDCoE. The obvious link between the data ‘science’ and a lab is on purpose since the environment should be a creative space to experiment and run test data analyses in order to achieve the desired results.

A well-designed Big Data lab contains open work-spaces that allow for communication and collaboration as well as isolated work possibilities where data analysts can ‘crunch the number’ without distractions.

A second important requirement for the Big Data labs is to have the hardware compatible for Big Data processing. In general, Big Data labs require hardware with sufficiently larger RAM than usual for Big Data processing.

Big Data Proof-of-Concepts

Proof-of-Concepts (POC) are showcase solutions that can be provided to internal business units as well as external clients. The POCs should demonstrate a clear return on investment and clearly showcase the capabilities of the Big Data Centre of Excellence in achieving the results.

Proof-of-Concepts are usually requested by internal business units or customers based upon specific Big Data questions (discussed in section 6.2, the first step of the Bid Data analysis process). By demonstrating clear Proof-of-Concepts for potential use cases, the BDCoE can showcase its knowledge in the enterprise.

Agile Methodology

Agility and the ability to fail fast or achieve quick results are essential to reaching the potential of Big Data. An Agile working methodology provides the tools to deliver outcomes quickly and transparently, typically within two- to three-week sprints. The ability to fail fast is a key Big Data opportunity ― business and technical roadmaps for delivering value need to change more often than in a traditional waterfall environment.

Charging Models

At the core of the Big Data Centre of Excellence are the charging models to justify the (sometimes large) investments in the people, processes and technology of the Centre. In order to display value, a clear approach needs to be devised to charge other business units or external clients for services rendered.

Charging models can be devised based on the number or users, data processed, frequency of reports or subscription based. A sound and unambiguous charging model will greatly help to showcase the value of BDCoE to the enterprise.