Applied statistics:

 

After a brief understanding of certain concepts now is the time to deep dive into the next topics related to the Applied statistics. To apply the concepts for the Statistics, we need data, from where can we get the data. Data sources are classified into Primary data and Secondary data.

 

 

Primary Data is collected continuously and Secondary data is one that is already collected, stored, and archived.

Primary data are collected by the organization itself for a particular purpose. The benefits of primary data are that they fit the needs exactly and are UpToDate and reliable.

 

Secondary data are collected by other organizations or for other purposes. Any data, which are not collected by the organization for a specific purpose are secondary data. These might be published by other organizations, available from research studies, published by the government, web, social media, and so on.

 

 

 

This brings us to another aspect of data which are subdivided into two i.e. Qualitative Data and Quantitative Data.

 

Qualitative data are non-numeric in nature and can't be measured. Examples are gender, religion, place of birth.

Quantitative data are numerical in nature and can be measured. Examples are balance in year savings bank account and a number of members in your family.

 

Quantitative data can be classified into discrete type or continuous type. Discrete type can take only certain values and there are discontinuities between values, such as the number of rooms in a hotel, which cannot be in fraction. Continuous type can take any value within a specific interval, such as the production quantity of a particular type of paper measured in Kg.

 

Any Dataset in Statistics or Data Science or Machine Learning has the below-mentioned features and classifications of the dataset.

 

Types of datasets:

1] Record:

Relational records

Data matrix, e.g. numerical matrix

Document Data, e.g. Text documents

Transaction data

 

2] Graph and Network:

World wide web

Social or information News

Molecular structures

 

3] Ordered:

Video data: Sequence of images

Temporal data: Time series

Sequential data: Transaction sequences

Genetic sequence data

 

4] Spatial, Image, and Multimedia:

Spatial Data: maps

Image data

Video Data

 

 

Data Objects:

Data sets are made up of data objects.

A Data object represents and entity.

Examples: Sales Database, University Database

Also called samples, examples, instances, data points, objects, tuples

Data Objects are described by attributes

Database rows: Data Objects

Database columns: Attributes

 

 

 

Attributes:

Attribute (or dimensions, features, variables) is a data field, representing a characteristic or feature of a data object.

e.g. customer ID, name, address

Types of attributes:

Nominal

Binary

Ordinal

Numeric(quantitative: Interval or scaled)

 

 

Attribute Types (Qualitative):

Nominal: Categories, States or "Name of things", hair color, marital status, occupation, ID number, Zipcodes, etc.

 

Binary: Nominal attribute with only 2 states. Symmetric binary, both outcomes equally important. e.g. gender. Asymmetric binary, both outcomes not equally important. Medical tests positive or negative

 

Ordinal: Values have meaningful order like ranking but the magnitude between successive values is not known.

 

 

Numeric Attribute (Quantitative):

Quantity: i.e. Integer or real valued number

 

Interval: Measured on a scale of equal-sized units, Values have order. No true zero points.

 

Ratio: Inherent zero point. We can speak of values as being an order of magnitude larger than a unit of measurement

 

 

 

Descriptive Statistics:

Points for discussion under descriptive statistics:

1] Raw Data

2] Frequency distribution- Histograms

3] Cumulative frequency Distribution

4] Measures of Central Tendency

5] Mean, Mode and Median

6] Measures of dispersion

7] Range, IQR, Standard Deviation, Co-efficient of variation

8] Normal Distribution, The empirical rule, Chebyshev rule

9] Five number summary

 

 

Data Versus Information:

When analysts are bewildered by a plethora of data, which do not make any sense on the surface of it, they are looking for methods to classify data that would convey meaning. The idea here is to help them draw the right conclusion. Data needs to be arranged to information.

 

1] Raw data:

Raw data represents numbers and facts in the original format in which the data have been collected. We need to convert the raw data into information for decision making

 

Data -----------------------------> Information ----------> Knowledge ------------> Wisdom

          Descriptive statistics

 

2] Frequency distribution:

In simple terms, the frequency distribution is a summarized table in which raw data are arranged into classes and frequencies. Frequency distribution focuses on classifying raw data into information. It is a widely used data reduction technique in descriptive statistics.

 

Histogram:

The histogram is a snapshot of the frequency distribution. The histogram is a graphical representation of the frequency distribution in which the X-axis represents data classes and the Y-axis represents the frequencies in bars.

Histogram depicts the pattern of distribution emerging from the characteristic being measured.

 

3] Cumulative frequency distribution:

A type of frequency distribution that shows how many observations are above or below the lower boundaries of the classes.

Example of the Cumulative frequency distribution:

 

4] What is Central Tendency?

Whenever you measure things of the same kind, a fairly large number of such measurements will tend to cluster around the middle value is called a measure of "Central Tendency".  In the Histogram figure the largest center frequency can be quoted as Central tendency.

 

 

5] Mean, Mode and Median:

Mean:

Mean or arithmetic mean is the simplest notion of central tendency.

Arithmetic mean is defined as the sum of all observations in a data set divided by the total number of observations.

X = summation of all values / number of observations

Arithmetic mean is affected by extreme values or fluctuations in sampling every high or low value

 

Mode: Mode is that value that occurs most often. It has the maximum frequency of occurrence. Mode also has resistance to outliers. Mode is a very useful measure when you want to keep in the inventory, the most popular shirt in terms of collar size during the festive season would be the mode.

 

Median:

Median is the middlemost observation when you arrange data in ascending order of magnitude. Median is such that 50% of observations are above the median and 50% of observations are below the median.

Median is a very useful measure for ranked data in the context of consumer preferences and rating. It is not affected by the extreme values (greater resistance to outliers)

Median = (number of observations + 1)/2 th value of ranked data

If one middle value then there is no problem, if we get two middle values then it would be the average of both.

 

 

 

6] Measures of dispersion:

In simple terms, measures of dispersion indicates how large the spread of distribution is around the central tendency. It answers unambiguously the question "What is the magnitude of departure from the average value for different groups having identical averages?"

 

 

7] Range, IQR, Standard Deviation, Co-efficient of variation:

Range:

The range is the simplest of all measures of dispersion. It is calculated as the difference between the maximum and minimum values in the dataset. Range = Xmax - Xmin

 

IQR (Inter Quartile range):

Range computed on middle 50% of the observations after eliminating the highest and lowest 25% of observations in a dataset that is arranged in ascending order. IQR is less affected by outliers.

IQR = Q3 - Q1

Q3 = Third or upper quartile

Q1 = First or lower quartile

 

 

Standard deviation:

To define Standard deviation we need to know the term variance.

In simple terms, Standard deviation is the square root of the variance.

Standard deviation looks at how spread out a group of numbers is from the mean, by looking at the square root of the variance.

The variance measures the average degree to which each point differs from the mean—the average of all data points.

The variance is the average of the squared differences from the mean.

 If n is the size of the population

 

 

Co-efficient of variation:

The coefficient of variation is defined as the ratio of Standard deviation to mean.

 

8] Normal distribution, The Empirical rule, Chebyshev rule:

Normal distribution:

The normal distribution is a continuous probability distribution that is symmetrical on both sides of the mean, so the right side of the center is a mirror image of the left side. It has a bell-shaped curve.

 

The Empirical rule:

The empirical rule approximates the variation of data in a bell-shaped distribution also known as a bell curve.

Approximately 68% of the data in bell-shaped distribution is within 1 std deviation of the mean.

Approximately 95% of data in a bell-shaped distribution lies within 2 std deviations of the mean.

Approximately 99.7% of data in a bell-shaped distribution lies within 3 std deviations of the mean.

 

 

Chebyshev rule:

This rule is used when the distribution is not normal or data is not in bell shape.

Regardless of how the data is distributed, or a wide class of probability distributions, no more than a certain fraction of values can be more than a certain distance from the mean. Specifically, no more than 1/k² of the distribution's values can be k or more standard deviations away from the mean (or equivalently, over 1 − 1/k² of the distribution's values are less than k standard deviations away from the mean). The rule is often called Chebyshev's theorem, about the range of standard deviations around the mean, in statistics.

We can get more depth about the differences between Empirical and Chebyshev's rule from the video below:

Chebyshev's rule vs empirical rule

  

9] Five number summary:

On a normal distribution five numbers describe center, spread, the shape of data:

Quartiles split the distribution in 4 equal parts.

1] Xsmallest: This is the smallest number from the normal plot.

 

2] First Quartile (Q1): The first quartile divides the smallest 25% of the values from the rest that is larger.

Q1 = (Number of observations + 1) / 4

 

2] Second Quartile (Q2): Second quartile which indeed is the median and divides 50% of the values from the rest that are larger or equal to the median.

Q2 = (Number of observations  + 1) / 2

 

3] Third Quartile (Q3): The third quartile divides the smallest 75% of the values from the rest that is larger.

Q3 = (3/4) * (Number of observations   + 1)

 

4] Xlargest: This is the largest number from the normal plot.

 

 

In the next blog, we will discuss more about Probability which is essential for Inferential Statistics.