My Notes Day 2

 Short overview of Data Science:

Due to increase in the mobiles, cloud data etc. the source of Data has increased a lot.

Evolution of technology, social media data, iot data has impacted a lot in the increase of data sources. Moreover the online shopping data, education, banking and finance, media digitalization has increased the overall impact.

Data Science focuses on processing and analysis of the data.

Data science is a process of extracting the knowledge and insights form the data by using scientific methods.

Data Science-> Programming + Statistics + Business

Torture the data and it will confess to anything.

 Data Scientist  must be good inn Math, technology and the business

Skillset required: Statistics, Programming languages, Data Extraction and processing, Data wrangling and extrapolation, Machine Learning, Big Data processing frameworks, Data Visualization.

Statistics: Math for understanding the data

Programming: R, Python with sophisticated algorithms and libraries

Data extraction, processing, Data Wrangling and exploration: Determine uninit data or inconsistent and missing data and then cleaning data and try making sense of data

ML: Algorithm to make machine understand to process the data.

Big data processing frameworks: No traditional data processing-> Hadoop to be used.

*Data lifecycle:

Let us now just look at some of the Software engineering basics.

Data structures and algorithms notes:

Big Oh Notation:

  • Used for time complexity analysis.
  • Provides an abstract measurement by which we can judge the performance of algorithms without using mathematical proofs.

  • Cubic and exponential algorithm should only be used for very small problems (if exists), mostly try to avoid.

  • O(1) constant: Operation doesn't depend on the input example adding a node at tail of linked list
  • O(n) linear: Runtime complexity is directly proportional to size of n
  • O(log n) logarithmic: Normally, associated with the algorithm that break the problem into smaller chunks per each invocation example binary search tree
  • O(n log n): Usually with an algorithm that breaks the problem into smaller chunks per each invocation, then takes the results of these smaller chunks and stiches them back together example, quick sort.
  • O(n^2): Quadratic, bubble sort
  • O(n^3): very rare
  • O(2^n): incredibly rare
Sometimes algorithm designs are done for solving problems irrespective of how it works. Strongly these algorithms should be reviewed to optimize where possible, particularly loops and recursive calls so that you can get most efficient run times for your algorithm.
Big Oh notation is important as irrespective of the hardware to be used the algorithm gives better runtime results. For example quadratic and logarithmic algorithms are available for a problem, for sure one will go with the logarithmic algorithm as the input number increases in the quadratic it tends to infinity. Time taken by the logarithmic algorithm is very very less.

Oops concepts: Inheritance, Encapsulation, polymorphism

Pseudocode

Comments

Post a Comment

Popular posts from this blog

Epoch 3: Solving problems by searching

Image
Let us continue with our journey further with the AI, a modern approach.   In this chapter we look into how an AI agent achieves a goal with a sequence of actions. These types of agents are also known as problems solving agents. Before this we will just look at the diagram below.   In an atomic representation each state of the world is indivisible, it has no internal structure. A factored representation splits up each state into a fixed set of variables or attributes, each of which can have a value. Structured representations underlie relational databases and first-order logic. The axis along which atomic, factored, and structured representations lie is the axis of increasing expressiveness. Roughly speaking, a more expressive representation can capture, at least as concisely, everything a less expressive one can capture, plus some more.   PROBLEM-SOLVING AGENTS   Whenever we have to solve a problem, what has to be achieved, has to be analyzed in the first plac...
Read more

My Notes Day 3

  Linked Lists: Series of nodes, each node has a single pointer to the next node and in the last node's case a null pointer representing that there are no more nodes in linked list. Linked list will always maintain head and tail pointers so that insertion at either head or tail of the list is a constant time operation. Random insertion is excluded from this and will be a linear operation. Insertion - O(1) Deletion - O(n) Searching - O(n) Here we chose to always maintain pointers to the nodes at the head and tail of the linked list and so performing a traditional insertion to either front or back of linked list is O(1) operation. An exception to this rule is performing insertion before a node that is neither the head nor the tail in a singly linked list. When the node we are inserting before is somewhere in the middle of the linked list (the random insertion) the complexity is O(n). In order to add before the designated node we need to traverse linked list to find that node's cu...
Read more

My notes Day 1

 ML Introduction: Data Science and Machine learning: Definition: Data Science:  Extracting hidden patterns from huge datasets. Patterns- Trends, association, hidden rules, clusters. example temperature rises ice-cream melts Science- Here it means statistics and mathematics not physical science Data sets usually refer to large volume of cleansed, structured data prepared for analysis. Science refers to statistical tools and techniques employed to understand data and reliability of the identified patterns. A] That part of statistics which is used to understand the data is called descriptive statistics.  Descriptive statistics gives vital insights into data in terms of central values, spread and distribution shape of data. B] Part of statistics which is used to establish the reliability of the potential patterns identified is called inferential statistics. Machine learning: Machine learning is an integral and critical part of Data science. It refers to a collection of algori...
Read more