My notes Day 1

 ML Introduction:

Data Science and Machine learning:

Definition:

Data Science: Extracting hidden patterns from huge datasets.

• Patterns- Trends, association, hidden rules, clusters. example temperature rises ice-cream melts
• Science- Here it means statistics and mathematics not physical science
• Data sets usually refer to large volume of cleansed, structured data prepared for analysis.
• Science refers to statistical tools and techniques employed to understand data and reliability of the identified patterns.
• A] That part of statistics which is used to understand the data is called descriptive statistics. Descriptive statistics gives vital insights into data in terms of central values, spread and distribution shape of data.
• B] Part of statistics which is used to establish the reliability of the potential patterns identified is called inferential statistics.

Machine learning:

• Machine learning is an integral and critical part of Data science. It refers to a collection of algorithms which are used to extract the hidden patterns from the dataset.
• These algorithms use a learning process through which they identify the patterns in the dataset. The patterns they learn from the data are called models.
• The models could be expressed in form of mathematical equations, rules, probability ratios etc.
• ML algorithms work on data prepared for analytics to express hidden patterns in form of models.
• For ML algorithms to successfully identify reliable hidden patterns the input data should be reliable.
• If input data is not reliable, models generated may be statically unreliable.

When is ML successful?

• Cannot express our knowledge about patterns as a program for example char recognition or Natural Language processing.
• Do not have an algorithm to identify a pattern of interest example in spam mail decision.
• Too complex and dynamic example weather forecasting
• No prior experience on knowledge for example Mass rover.
• Patterns hidden in humongous data  for example recommendation system.

ML applications:

• Fraud detection
• Sentiment Analysis
• Credit risk management 
• Prediction of equipment failures
• New pricing models/strategies
• Network intrusion decision
• Pattern and image recognition
• Email spam filtering

 ML prerequisites:

• Rich set of data representing the environment where the model is to be used.
• Knowledge and skills in Mathematics, Statistics, Programming language Python and R, Domain Knowledge
• Usually data science is a team effort where team consists of all required skills and knowledge

ML Categories:

1. Supervised learning
2. Unsupervised learning
3. Reinforcement Learning

Supervised Machine Learning:

• Class of algorithms which work in 2 stages. First stage is called training and second one is usually called testing. Sometimes it may involve validation stage followed by testing.
• At each stage it takes input data prepared for that stage, i.e., for training data  stage, test data for test stage, validation data for validation stage.
• During training, the Machine learning algorithm gets the training data in form of independent and dependent variables.
• In the process of learning, the algorithm learns the relationship between dependent and independent variables.
• This relationship is expressed as a model which can take the form of equation, probability ratios hidden rules etc.
• Supervised ML can be further classifies into regression (when predicting numeric values) or classification (when predicting class labels)

Examples of supervised ML:

1. Regression: Predicting mileage of a car given the other features such as weight, engine capacity, horse power, transmission type, number of cylinders etc. In this example mileage is dependent variable and weight, engine capacity, hp, transmission type, number of cylinders are independent variables. Mileage = f(weight, engine capacity, horsepower, transmission type, number of cylinder)
2. Classification: Categorizing a mail into spam or ham. In this example the email category is a target variable (spam or ham) and occurrences of certain words and their frequency are independent variables. P(ham) = f(word, frequencies) where P stands for probability . 1 = P(ham) will give the P(spam) assuming only 2 categories ham and spam.

Unsupervised ML:

• Class of algorithm which work in a single stage. Unlike supervised learning algorithms, it doesn't have a separate training, testing or validation stage.
• Unsupervised learning algorithms take the data as a whole not in form of independent and dependent variables.
• Algorithms are not used to find any relationship between dependent and independent variables.
• This class of algorithms usually find pattern in form of clusters and associations reflecting some kind of commonality, togetherness among data points in given datasets.
• It is responsibility of a data scientist to analyze the identified clusters associations and give meaning to those clusters
• Clustering and Principal component analysis i.e., PCA a mathematical technique used to transform given data into more useful form, belong to this category of ML. 

Examples of unsupervised ML:

• Clustering- Identifying groups in given data set where a group represents some kind of commonality among the data points.
• Clustering can further be categorized into Flat clustering and Hierarchical clustering. 
• Flat clustering: example K means clustering- The clusters identified are disjoint, non-overlapping. For example segmenting customers into different groups based on their purchase amount, frequency of purchase and types of items purchase.
• Hierarchical clustering: Clustering is done at multiple levels indicating clusters inside clusters indicating some kind of subgroups inside a given group. For example to identify sub-clusters within cash-cow customers from K means.

Reinforced ML:

• Reinforcement learning algorithm learns through trial and error and the feedback it receives from the environment in which it learns
• During the initial stages of learning it is likely to commit many errors in learning patterns, however through a process of reward and punishment's it learns to identify patterns correctly.
• Self driving cars is an example of reinforced learning.