The Big Data Framework

Why a Big Data Framework?

Frameworks provide structure. The core objective of the Big Data Framework is to provide a structure for enterprise organisations that aim to benefit from the potential of Big Data. In order to achieve long-term success, Big Data is more than just the combination of skilled people and technology – it requires structure and capabilities.

The Big Data Framework was developed because – although the benefits and business cases of Big Data are apparent – many organizations struggle to embed a successful Big Data practice in their organization. The structure provided by the Big Data Framework provides an approach for organizations that takes into account all organizational capabilities of a successful Big Data practice. All the way from the definition of a Big Data strategy, to the technical tools and capabilities an organization should have.

The main benefits of applying a Big Data framework include:

1. The Big Data Framework provides a structure for organisations that want to start with Big Data or aim to develop their Big Data capabilities further.
2. The Big Data Framework includes all organisational aspects that should be taken into account in a Big Data organization.
3. The Big Data Framework is vendor independent. It can be applied to any organization regardless of choice of technology, specialisation or tools.
4. The Big Data Framework provides a common reference model that can be used across departmental functions or country boundaries.
5. The Big Data Framework identifies core and measurable capabilities in each of its six domains so that the organization can develop over time.

Big Data is a people business. Even with the most advanced computers and processors in the world, organisations will not be successful without the appropriate knowledge and skills. The Big Data Framework therefore aims to increase the knowledge of everyone who is interested in Big Data. The modular approach and accompanying certification scheme aims to develop knowledge about Big Data in a similar structured fashion.

The Big Data framework provides a holistic structure toward Big Data. It looks at the various components that enterprises should consider while setting up their Big Data organization. Every element of the framework is of equal importance and organisations can only develop further if they provide equal attention and effort to all elements of the Big Data framework.

The Structure of the Big Data Framework

The Big Data framework is a structured approach that consists of six core capabilities that organisations need to take into consideration when setting up their Big Data organization. The Big Data Framework is depicted in the figure below:

The Big Data Framework consists of the following six main elements:

1. Big Data Strategy

Data has become a strategic asset for most organisations. The capability to analyse large data sets and discern pattern in the data can provide organisations with a competitive advantage. Netflix, for example, looks at user behaviour in deciding what movies or series to produce. Alibaba, the Chinese sourcing platform, became one of the global giants by identifying which suppliers to loan money and recommend on their platform. Big Data has become Big Business.

In order to achieve tangible results from investments in Big Data, enterprise organisations need a sound Big Data strategy. How can return on investments be realised, and where to focus effort in Big Data analysis and analytics? The possibilities to analyse are literally endless and organisations can easily get lost in the zettabytes of data. A sound and structured Big Data strategy is the first step to Big Data success.

2. Big Data Architecture

In order to work with massive data sets, organisations should have the capabilities to store and process large quantities of data. In order to achieve this, the enterprise should have the underlying IT infrastructure to facilitate Big Data. Enterprises should therefore have a comprehensive Big Data architecture to facilitate Big Data analysis. How should enterprises design and set up their architecture to facilitate Big Data? And what are the requirements from a storage and processing perspective?

The Big Data Architecture element of the Big Data Framework considers the technical capabilities of Big Data environments. It discusses the various roles that are present within a Big Data Architecture and looks at the best practices for design. In line with the vendor-independent structure of the Framework, this section will consider the Big Data reference architecture of the National Institute of Standards and Technology (NIST).

3. Big Data Algorithms

A fundamental capability of working with data is to have a thorough understanding of statistics and algorithms. Big Data professionals therefore need to have a solid background in statistics and algorithms to deduct insights from data. Algorithms are unambiguous specifications of how to solve a class of problems. Algorithms can perform calculations, data processing and automated reasoning tasks. By applying algorithms to large volumes of data, valuable knowledge and insights can be obtained.

The Big Data algorithms element of the framework focuses on the (technical) capabilities of everyone who aspires to work with Big Data. It aims to build a solid foundation that includes basic statistical operations and provides an introduction to different classes of algorithms.

4. Big Data Processes

In order to make Big Data successful in enterprise organization, it is necessary to consider more than just the skills and technology. Processes can help enterprises to focus their direction. Processes bring structure, measurable steps and can be effectively managed on a day-to-day basis. Additionally, processes embed Big Data expertise within the organization by following similar procedures and steps, embedding it as ‘a practice’ of the organization. Analysis becomes less dependent on individuals and thereby, greatly enhancing the chances of capturing value in the long term.

5. Big Data Functions

Big Data functions are concerned with the organisational aspects of managing Big Data in enterprises. This element of the Big Data framework addresses how organisations can structure themselves to set up Big Data roles and discusses roles and responsibilities in Big Data organisations. Organisational culture, organisational structures and job roles have a large impact on the success of Big Data initiatives. We will therefore review some ‘best practices’ in setting up enterprise big data

In the Big Data Functions section of the Big Data Framework, the non-technical aspects of Big Data are covered. You will learn how to set up a Big Data Center of Excellence (BDCoE). Additionally, it also addresses critical success factors for starting Big Data project in the organization.

6. Artificial Intelligence

The last element of the Big Data Framework addresses Artificial Intelligence (AI). One of the major areas of interest in the world today, AI provides a whole world of potential. In this part of the framework, we address the relation between Big Data and Artificial Intelligence and outline key characteristics of AI.

Many organisations are keen to start Artificial Intelligence projects, but most are unsure where to start their journey. The Big Data Framework takes a functional view of AI in the context of bringing business benefits to enterprise organisations. The last section of the framework therefore showcases how AI follows as a logical next step for organisations that have built up the other capabilities of the Big Data Framework. The last element of the Big Data Framework has been depicted as a lifecycle on purposes. Artificial Intelligence can start to continuously learn from the Big Data in the organization in order to provide long lasting value.

Learn more about the Big Data Framework

This article is an excerpt out the Enterprise Big Data Professional guide. If you want to learn more about the Big Data Framework, you can download a copy of the guide for free on this page.