Introduction to Applied Statistics for Data Science

 

With the increase in the use of devices such as smartphones, apps, cloud, social media, online shopping, education, banking, etc. where more and more technology is expected to serve mankind statistics plays a key role in the area of science, finance, and industries (Right from Software to Construction industries). Processing and analyzing user data is very important. Predicting a patient's health who can from cardiac arrest, or monitoring the ones which have already undergone through it. Predicting the price of a stock in 6 months in the future, based on past and current financial performance. Identify the handwritten numbers. To actually dive deep into Data Science, Machine Learning, and Artificial Intelligence we need some insights into Statistics, Matrices, Vectors, and Probability. Some knowledge about the Differential Equations, Curve fitting, etc. would be an add-on. Mathematics is the mother of these concepts. We will look into these as pre-requisites

 

 

Why is there a sudden increase in trends like Data Science, Machine Learning, Statistics?

Answer: Three significant events triggered current meteoric growth in the use of analytical decision making and statistics being center for all of it.

Event1: Technological developments, the revolution of internet and social networks, data generated from mobile phones and other electronic devices, produce a large amount of data from which insights have to be classified.

Discovery of pattern and trends from these data for organizations will prove the way for improving profitability, understanding customer expectations, and appropriately pricing their products so that they can gain a competitive advantage in the market place

Event2: Advances in enormous computing power to effectively process and analyze massive amounts of data. When a design is not available, statistics help to interpret and extract the solutions for the same.

Sophisticated and faster algorithms for solving problems.

Data visualization for Business Intelligence and Artificial Intelligence

Event3: Large data storage capability, parallel computing and cloud computing coupled with better Computer Hardware has enabled the businesses and other organizations to solve large scale problems faster than ever before without sacrificing

 

 

Big Data: A set of data that cannot be managed, processed or analyzed with traditional Software or algorithms within a reasonable amount of time

Big data revolves around Volume, Velocity, Value

This Data has to be used for many applications which is indeed a study of Data Science.

Data sources and types of datasets and their attributes are of most concern for the study of data science or statistics

There are certain types of statistics and statistical methods.

 

 

Statistical methods:

1] Classification

2] Pattern recognition

3] Association

4] Predictive Modelling

 

1] Classification:

Classification techniques help in segmenting the customers into appropriate groups based on key characteristics.

For example, using an appropriate statistical model, an organization could easily segment the customers into long term customers, medium-term customers and Brand switchers

Another application in the context is classifying customers into "Buyers and Non-Buyers".

Classification helps professionals understand the customer behavior and position their products and brands using appropriate strategies.

Consider Sentiment Analysis(+ve or -ve): Analysis depending on the customer feedback if he/she likes the service. If one can classify then we can do sentiment analysis or other analysis which are part of classification techniques.

 

2] Pattern recognition:

Analyzing different data in form of a pattern, "A picture is worth a thousand words" and it reveals a hidden pattern in the data that could be leveraged by retail professionals. Pattern recognition techniques include Histogram, Box plot, scatter plot, and other visual analytics. For example, a histogram drawn for the income of a particular class of customers may reveal, a symmetrical bell curve pattern or maybe left or right-skewed.

The relationship between age and expenditure can be captured using a scatter plot. Box plot enables identification of outliers (extreme positions) apart from providing the distribution patterns

 

 

3] Association:

Co-relation in simple terms, Association analysis helps in determining which of the items go together. Association rules include a set of analytics that focuses on discovering relationships that exist among specific objects.

In this context, market basket analysis refers to an association rule that generates the probability for an outcome. For example, market basket analysis may lead to a finding that if customers buy coffee, there is a 40% chance that they would also buy bread.

Association rules can be adopted by organizations to the store layout, items handling, discount, and sales promotion decisions, and cross selling among others.

 

4] Predictive modeling:

Both customer segmentation as well as identifying and targeting the most profitable customers can be facilitated by predictive models. Regression can be used for predicting the amount of expenditure on a particular product, based on input variables, income, age, and gender.

Organizations can leverage other advanced models that comprise logistic regression and neural networks for predicting a target variable as well as classifying and predicting into which group the customer belongs to. For example, these models can classify and predict buyers, non-buyers, defaulters, non-defaulters on credit cards and loans.

 

The classical definition of Statistics:

By Statistics, we mean methods specially adapted to the elucidation of quantitative data affected to a marked extent by a multiplicity of causes.

Business analysis can be defined as the broad use of data and quantitative analysis for decision-making within organizations.

 

Types of Statistics:

1] Descriptive Statistics:

It is concerned with Data summarization, graphs/charts, and tables or data which show what (what is or) has already happened.

2] Inferential Statistics:

A method used to talk about a population parameter (what might be) from a sample. What might happen, to draw a certain conclusion.

 

Describe -> Infer -> Predict

 

Population, Parameters, Sample, Statistic:

 

Data Now ---> Conclusions  ----> Future Data

 

Data Now <----> Population <-----> Future Data

 

Population: Universe of possible data for a specified object. Example: People who visited a blog

 

Parameter: Numerical value associated with a population. Example: Average amount of time people spent on the blog

 

Sample: Selection of observations from a population. Example: IP address of people who visited blog on a specific day

 

Statistic: Numerical value associated with an observed sample. Example: Average amount of time-specific IP address person spent on a specific day

 

Sample and Statistics can be observed but population and the parameter cannot be observed.

 

Some interesting videos on how retailers make customer buy more:

What ideas do retailers use for more sales: Link

 

Supermarket strategy in Hindi: Link

 

How stores track your behavior: Link

 

Upcoming next more details of descriptive statistics, Qualitative and Quantitative data.