Applied Statistics: Probability and Distributions

As we continue our journey towards data science, let us create a firm base for statistics, also let us create a good understanding of some of the concepts which will be very much in use when we deal with the data. We saw some details about the probability theory in our previous discussions, let us now get into the details of Joint, Marginal, and Conditional probability.

Marginal probability: the probability of an event occurring (P(A)), it may be thought of as an unconditional probability.  It is not conditioned on another event

 

Joint probability:  P(A and B).  The probability of event A and event B occurring.  It is the probability of the intersection of two or more events.  The probability of the intersection of A and B may be written P(A ∩ B).

 

Conditional probability:  P(A|B) is the probability of event A occurring, given that event B occurs. 

 

The equation below is a means to manipulate among joint, conditional, and marginal probabilities.  As you can see in the equation, the conditional probability of A given B is equal to the joint probability of A and B divided by the marginal of B.

 

P(A|B) = P(A ∩ B) / P(A)

 

Bayes’ Theorem:

Used to revise previously calculated probabilities based on new information. It is an extension of conditional probability. Given a hypothesis H and evidence E, Bayes theorem states that the relationship between the probability of hypothesis P(H) before getting the evidence and the probability P(H|E) of the hypothesis after getting the evidence is

P(H|E) = P(E|H) P(H) / P(E)

Many modern Machine learning techniques rely on Bayes’ theorem. For example, mail spam filtering uses Bayesian updating to determine whether mail is real or spam by given words in an email. Additionally, many specific techniques in statistics such as calculating P-values or interpreting medical results, are best described in terms of how they contribute to update hypothesis using Bayes’ theorem.

Bayesian spam filtering example:

Event A: The message is spam

Test X: Message contains certain words (X)

Pr(spam|words) = Pr(words|spam) Pr(spam) / Pr(words)

Pr(spam): It will be updated from how many mails from total mails are spam

Pr(words): It will be updated from dictionary or the data, given to PC

Pr(words|spam): Determined from above probabilities

 



 

Bayesian filtering allows us to predict the chance, a message is really spam given the “test results” (the presence of certain words). Clearly, words like “Viagra” have a higher chance of appearing in spam messages than in normal ones.

Spam filtering based on a blacklist is flawed, it is too restrictive, and false positives are too great. But Bayesian filtering gives us a middle ground. We use probabilities. As we analyze the words in a message, we can compute the chance if it is spam (rather than making a yes/no decision). If a message has a 99.9% chance of being spam, it probably is. As the filter gets trained with more and more messages, it updates the probabilities that certain words lead to spam messages. Advanced Bayesian filters can examine multiple words in a row, as another data point.

 

Distributions: A Distribution is a collection of probabilities that apply to a specific random situation.

What is a probability distribution?

In precise terms, a probability distribution is a total listing of various values a random value can take along with the corresponding value of each value. A real-life example could be the pattern of distribution of machine breakdowns in a manufacturing unit. The random value in this example would be various values the machine breakdowns could assume. The probability corresponding to each value of breakdowns is a relative frequency of occurrence of breakdown. The probability distributions, for this example is, constructed by, actual breakdown pattern observed over a period of time. Statisticians use the term “observed distribution” of breakdown.


 


Binomial Distribution:

The Binomial distribution is a widely used probability distribution of a discrete random variable. It plays a major role in quality control and quality assurance function. Manufacturing units do use the binomial distribution for defective analysis. Reducing the number of defectives using the proportion defective control chart (p chart) is an accepted practice in manufacturing organizations. It is also used in service organizations like banks, and insurance organizations to get an idea of the proportion of customers who are satisfied with the service quality.

Conditions for applying Binomial distribution (Bernoulli’s Process):

  1. Trails are independent and random
  2. There are a fixed number of trials (n trials)
  3. There are only 2 outcomes of trials designated as success or failure
  4. The probability of success is uniform throughout n trials



Binomial Probability function: Under the conditions of Bernoulli’s process. The probability of getting x success out on trials is indeed the definition of Binomial distribution. The Binomial probability function is given by the Binomial distribution formula as shown in the above diagram. P(x) is a probability of getting x successes in n trials, nCx is the number of ways in which x successes can take place out of n trials i.e. n!/x!(n-x)!

“p” is the probability of success which is the same throughout the n trials. “P” is the parameter of the binomial distribution, x can take values from 0, 1, 2….. n.

Example for Binomial distribution: A bank issues credit cards to customers under the scheme of master card. Based on the past data, the bank has found out that 60% of all accounts pay on time following the bill. If a sample of 7 accounts is selected at random from the current database, construct the binomial probability distribution of accounts paying on time (for 2 customers)

Solution:

P = 0.6, P(2) = (7C2)*(0.6)^2 *(1-0.6)^(7-2) = (7!/2!(7-2)!)*0.6*0.6*(0.4)^5 = 0.0774144

 

Mean and Standard deviation of Binomial distribution:



 

 

Poisson Distribution:

Poisson distribution is another discrete distribution, which also plays a major role in quality control in the context of reducing the number of defects per standard unit. Examples include the number of defects per item, number of defects per transformer produced, number of defects per 100 meter square of cloth etc. Other examples include the number of cars arriving at a highway check post per hour, Number of customers visiting a bank per hour during peak business period and number of pixels in an image, that are corrupted.

 



 

Example of Poisson distribution: If on an average, 6 customers arrive every 2 minutes at a bank during hours of work, What is the probability that exactly 4 customers arrive at a given minute?

Solution: P(x) = (e^-3)*^(3^4) / 4! = 0.1680

 

 



 

Normal distribution:

The empirical rule approximates the variation of data in a bell-shaped distribution. Approximately, 68% of data in a bell-shaped distribution is within 1 standard deviation of the mean. Approximately, 95% of the data in a bell-shaped distribution lies within 2 standard deviations of the mean. Approximately 99.7% of the data in a bell-shaped distribution lies within 3 standard deviations of the mean.

Properties of Normal distribution:

  • The normal distribution is a continuous distribution looking like a bell. Statisticians use the expression Bell-shaped distribution.
  • It is a beautiful distribution in which the mean, the median, and the mode are all equal to one another.
  • It is symmetrical about its mean.
  • If the tails of the normal distribution, are extended, they will run parallel to the horizontal axis, without actually touching it. (asymptotic to the x-axis)
  • The normal distribution has 2 parameters namely the men and the standard deviation.

 

Normal Probability Density function:

In usual notation, the probability density function of the normal distribution is given by:



 

X is a continuous normal random variable with the property, which means x can take all real numbers in the interval.

We will look into a python code, which would give a good idea about the above discussed Statistical concepts. Using the Jupyter notebook from the Anaconda Navigator.

Input csv file (the data set): 

Automobile_csv

Visualizing the dataset:

 Just starting with the data and looking into the various plots











Binomial distribution problem as discussed in the theory section:

A bank issues credit cards to its customers under the scheme of Master card. Based on past data, it is known that 60% of the customers pay the bill on time. If a sample of 7 accounts is selected at random from the current database, construct the binomial probability distribution of accounts paying on time.





 

Now let us look into Poisson distribution problem:

Given that a bank has a footfall of 6 people per minute, find out the

a) probability distribution for up to 20 people per minute
b) probability of at least 4 people per minute

 

 







 

This brings to the end of this section stay tuned for next Inferential statistics details.




Comments

Post a Comment

Popular posts from this blog

Epoch 3: Solving problems by searching

Image
Let us continue with our journey further with the AI, a modern approach.   In this chapter we look into how an AI agent achieves a goal with a sequence of actions. These types of agents are also known as problems solving agents. Before this we will just look at the diagram below.   In an atomic representation each state of the world is indivisible, it has no internal structure. A factored representation splits up each state into a fixed set of variables or attributes, each of which can have a value. Structured representations underlie relational databases and first-order logic. The axis along which atomic, factored, and structured representations lie is the axis of increasing expressiveness. Roughly speaking, a more expressive representation can capture, at least as concisely, everything a less expressive one can capture, plus some more.   PROBLEM-SOLVING AGENTS   Whenever we have to solve a problem, what has to be achieved, has to be analyzed in the first plac...
Read more

My Notes Day 3

  Linked Lists: Series of nodes, each node has a single pointer to the next node and in the last node's case a null pointer representing that there are no more nodes in linked list. Linked list will always maintain head and tail pointers so that insertion at either head or tail of the list is a constant time operation. Random insertion is excluded from this and will be a linear operation. Insertion - O(1) Deletion - O(n) Searching - O(n) Here we chose to always maintain pointers to the nodes at the head and tail of the linked list and so performing a traditional insertion to either front or back of linked list is O(1) operation. An exception to this rule is performing insertion before a node that is neither the head nor the tail in a singly linked list. When the node we are inserting before is somewhere in the middle of the linked list (the random insertion) the complexity is O(n). In order to add before the designated node we need to traverse linked list to find that node's cu...
Read more

My notes Day 1

 ML Introduction: Data Science and Machine learning: Definition: Data Science:  Extracting hidden patterns from huge datasets. Patterns- Trends, association, hidden rules, clusters. example temperature rises ice-cream melts Science- Here it means statistics and mathematics not physical science Data sets usually refer to large volume of cleansed, structured data prepared for analysis. Science refers to statistical tools and techniques employed to understand data and reliability of the identified patterns. A] That part of statistics which is used to understand the data is called descriptive statistics.  Descriptive statistics gives vital insights into data in terms of central values, spread and distribution shape of data. B] Part of statistics which is used to establish the reliability of the potential patterns identified is called inferential statistics. Machine learning: Machine learning is an integral and critical part of Data science. It refers to a collection of algori...
Read more