Big Data Issues

Big Data

Big Data Issues 

Big Data - A need to make fast decisions and act quickly act on the data information gained.
1  Data integration is the key
Data integration is absolutely essential for obtaing the best from your big data. Data integration addresses the need for removing data silos so that companies can obtain deeper insight from big data.
You should design your big data architecture for integration and governance from the very beginning.
Not only will this save the company the heavy workload that is inevitable when working with big data and data silos. It will also help to establish accuracy.
It will increase the trustworthiness of your data, which will cement the authority of any knowledge that you gain from analysing your data.
Data scientists have to clean the data before they can even process  it. This takes up to 80% of the work load. They are data custodians as well as analysts. If you have to repeat a task more than three times, you should automate it.
It will probably take longer the first time but then you will save time and focus on analysis from then onwards.
2.  The data silos problem
Data silos store all of that variety of data that you have captured in separate disparate units, that do not have anything to do with each other and therefore insights can not be gathered from this data  - because it is not integrated.
To produce a monthly sales report silos have to assimilate all of the numbers. It is a slow process that cause C-level decisions being made much to slowly.
Data silos are the reason your sales and marketing teams simply don’t get along. They are the reason that your customers are look elsewhere to take their business because they do not think that their needs are being met, and other more efficient  companies are offering something better.
The best way to eliminate data silos is to integrate your data. Not only are data silos ineffective on an operational level - but they provide inaccurate data which is greatest Big Data problem.
Inaccurate customer data is worse than no data at all. Eliminate data silos by integrating your data.
3  Separate the signal from the noise!
There has to be a discernible signal in the noise that you can detect, and sometimes there isn’t one to be able to use big data properly
Maybe we just didn’t measure correctly or measured the wrong variables because there's nothing we can detect here."
Therefore one of the biggest issues faced by businesses when handling big data is a like trying to find a needle in a haystack. A scientific approach to the data is necessary.
You need to approach it very carefully and like a scientist, so that means if you fail at your hypothesis, you come up with a few other hypotheses, and maybe one of them turns out to be correct."
data is an advantage SMEs have over large corporations.
4. The shortage of skilled workers
CapGemini's report found that 37% of companies have trouble finding skilled data analysts to make use of their data. Ideally it is best to form just one data analysis team for the company, either through re-training your current workers or recruiting big data specialists.
You need to find employees that understand both data from a scientific viewpoint spective, but who also understand the company business as well as its customers requirements -  and how their data analysis results are relevant to them.
5. Technology moves so fast.
Larger companies are more likely to suffer because of data silos, because they prefer to keep their databases on site and also because decision making about new technologies is typically slow.
An example mentioned in the CapGemini report is that large corporations like telcos and utilities getting high levels of disruption from new competitors moving in from other sectors. 
Typically traditional players are slower to adopt new technologies and are therrfore finding themselves faced with serious competition from smaller companies.
If you can obtain all the relevant data, analyze it quickly, surface actionable insights, and drive them back into operational systems, then you can affect events as they’re still occurring.

Big Data Trends

More organisations are storing, processing, and extracting value from data of all forms and sizes. Systems that support large volumes of both structured and unstructured data will continue to rise.

The market will demand platforms that help data custodians govern and secure big data while empowering end users to analyse that data.

These systems will mature to operate well inside of enterprise IT systems and standards. Take a look at what we anticipate to see with Big Data in the coming year.

1. Big data becomes faster and more flexible

Options expand to speed up Hadoop - you can perform machine learning and conduct sentiment analysis on Hadoop, but one of the first questions people often ask is: How fast is the interactive SQL?

The need for speed has driven the adoption of - Faster databases like Exasol and MemSQL, Hadoop-based stores like Kudu.

Faster queries can be performed using SQL-onHadoop engines (Apache Impala, Hive LLAP, Presto, Phoenix, and Drill) and OLAP-on-Hadoop technologies (AtScale, Jethro Data, and Kyvos Insights).

These query accelerators are blurring the lines between traditional warehouses and the world of big data.

2. Big data is now more than Hadoop:

Earlier purpose built tools for Hadoop have become obsolete.

Enterprises with complex, heterogeneous environments no longer want to adopt a siloed BI access point just for one data source – Hadoop.

Even relational databases are becoming big data-ready. SQL Server, for instance, recently added JSON support.

3. Organisations leverage data lakes in many ways

A data lake is like a man-made reservoir. First you dam the end (build a cluster), then you let it fill up with water (data). Once you establish the lake, you start using the water (data) for various purposes like generating electricity, drinking, and recreating (predictive analytics, predictive analytics, ML, cyber security, etc.

Up until now, hydrating the lake has been an end in itself. This will change as the business justification for Hadoop tightens. Organisations will demand repeatable and agile use of the lake for quicker answers. They’ll carefully consider business outcomes before investing in personnel, data, and infrastructure. This will foster a stronger partnership between the business and IT. And self-service platforms will gain deeper recognition as the tool for harnessing big-data assets.

4. Architectures mature to reject one-size-fits all frameworks

Hadoop is no longer just a batch-processing platform for data-science use cases. It has become a multi-purpose engine for ad hoc analysis. It’s even being used for operational reporting on day-to-day workloads — the kind traditionally handled by data warehouses. Organisations are responding to these hybrid needs by pursuing use case-specific architecture design.

Companies research a host of factors including user personas, questions, volumes, frequency of access, speed of data, and level of aggregation before committing to a data strategy. These modern-reference architectures will be needs-driven. They’ll combine the best self-service data-prep tools, Hadoop Core, and end-user analytics platforms in ways that can be reconfigured as those needs evolve. The flexibility of these architectures will ultimately drive technology choices.

5. Variety, not volume or velocity, drives big-data investments

While all three Vs are growing, variety is becoming the single biggest driver of big-data investments, as firms seek to integrate more sources and focus on the “long tail” of big data.

From schema-free JSON to nested types in other databases (relational and NoSQL), to non-flat data (Avro, Parquet, XML), data formats are multiplying and connectors are becoming crucial.

Companies will continue to evaluate analytics platforms based on their ability to provide live direct connectivity to these disparate sources.

Platforms that are data- and source-agnostic will thrive.

6. Spark and machine learning light up big data Apache Spark, once a component of the Hadoop ecosystem, is now becoming the big-data platform of choice for enterprises

In a survey of data architects, IT managers, and BI analysts, nearly 70% of the respondents favoured Spark over incumbent MapReduce, which is batch-oriented and doesn’t lend itself to interactive applications or real-time stream processing.

These big-compute-on-big-data capabilities have elevated platforms featuring computation-intensive machine learning, AI, and graph algorithms. Microsoft Azure ML in particular has taken off thanks to its beginner-friendliness and easy integration with existing Microsoft platforms.

Opening up ML to the masses will lead to the creation of more models and applications.

7. The convergence of IoT, cloud, and big data create new opportunities for self-service analytics. Everything will have a sensor that sends information back to the mothership

IoT is generating massive volumes of structured and unstructured data, and an increasing share of this data is being deployed on cloud services.

The data is often heterogeneous and lives across multiple relational and non-relational systems, from Hadoop clusters to NoSQL databases.

As a result, demand is growing for analytical tools that seamlessly connect to and combine a wide variety of cloud-hosted data sources. Such tools enable businesses to explore and visualise any type of data stored anywhere, helping them discover hidden opportunity in their IoT investment.

8. Self-service data prep becomes mainstream as end users begin to shape big data

Self-service data prep becomes mainstream as end users begin to shape big data making Hadoop data accessible to business users is one of the biggest challenges of our time. The rise of self-service analytics platforms has improved this journey.

But business users want to further reduce the time and complexity of preparing data for analysis, which is especially important when dealing with a variety of data types and formats.

Agile self-service data-prep tools not only allow Hadoop data to be prepped at the source but also make the data available as snapshots for faster and easier exploration.

Companies focused on end user data prep for big data such as Alteryx, Trifacta, and Paxata. These tools are lowering the barriers to entry for late Hadoop adopters and laggards and will continue to gain traction.

9. We are seeing a growing trend of Hadoop becoming a core part of the enterprise IT landscape

Investments in the security and governance components surrounding enterprise systems will rise.

Apache Sentry provides a system for enforcing fine grained, role-based authorisation to data and metadata stored on a Hadoop cluster.

Apache Atlas, created as part of the data governance initiative, empowers organisations to apply consistent data classification across the data ecosystem.

Apache Ranger provides centralised security administration for Hadoop.

Customers are starting to expect these types of capabilities from their enterprise-grade RDBMS platforms.

10. Increasingly metadata catalogs help people find analysis-worthy big data

For a long time, companies threw away data because they had too much to process.

With Hadoop, they can process lots of data, but the data isn’t generally organised in a way that can be found.

Metadata catalogs can help users discover and understand relevant data is worth analysing using self-service tools.

This gap in customer need is being filled by companies like Alation and Waterline - which use machine learning to automate the work of finding data in Hadoop.

They catalog files using tags, uncover relationships between data assets, and even provide query suggestions via searchable UIs.

This helps both data consumers and data stewards reduce the time it takes to trust, find, and accurately query the data.

In the coming year, we’ll see more awareness and demand for self-service discovery, which will grow as a natural extension of self service analytics.

Big Data and Cyber Security are Coliding!

Big data has begun to impact enterprise security.

How can companies handle the mass of security-relevant data to improve their security?

How can organisations actually secure their big data?

Security is a key part of a trusted IT infrastructure. Enterprises are developing their security strategy towards an intelligence driven model.

Response to security incidents is not just about technology, it’s about having the necessary skilled resources etc. Companies need a detailed security management strategy designed to protect against the latest threats, to provide visibility into the business risks, and to provide fast and accurate responses.

Big data security analytics and Cyber Intelligence have vitally necessary tools to combat cyber-attacks.

As enterprises embrace virtualisation, it is essential that they are completely aware of what is happening in their company and then to use effective security mechanisms.

Client Enquiry

Knight Hamilton provide senior executives and managers as well as key sales, marketing and technical candidates.

Find Out More

Candidate Enquiry

Knight Hamilton provide a range of candidate services.

We have continual vacancies for managerial, technical and general job opportunities.

Find Out More