TYBSC-CS-Data-science-munotes

Page 1

1 1
INTRODUCTION TO DATA SCIENCE
Unit Structure
1.1 What is Data?
1.2 Different kinds of data
1.3 Introduction to high level programming language
1.4 Integrated Development Environment (IDE)
1.5 Exploratory Data Analysis (EDA)
1.6 Data Visualization
1.7 Different types of data sources
1.8 Summary
1.9 Questions
1.1 WHAT IS DATA?
In general, data is a distinct piece of information that is gathered and
translated for some purpose. If data is not formatted in a specific way, it
does not valuable to computers or humans. Data can be available in term s
of different forms, such as bits and bytes stored in electronic memory,
numbers or text on pieces of paper, or facts stored in a person's mind.
Since the invention of computers, people have used the word data to mean
computer information, and this inform ation is transmitted or stored. There
are different kinds of data; such are as follows:
Sound
Video
Single character
Number (integer or floating -point)
Picture
Boolean (true or false)
Text (string)
In a computer's storage, data is stored in the form of a s eries of binary
digits (bits) that contain the value 1 or 0. The information can be in terms
of pictures, text documents, software programs, audio or video clips, or
other kinds of data. The computer data may be stored in files and folders
on the computer' s storage, and processed by the computer's CPU, which
utilizes logical operations to generate output (new data) form input data. munotes.in

Page 2

Data science
2 As the data is stored on the computer in binary form (zero or one), which
can be processed, created, saved, and stored digitall y. This allows data to
be sent from one computer to another with the help of various media
devices or a network connection. Furthermore, if you use data multiple
times, it does not deteriorate over time or lose quality.
Examples of computer data
0143 0157 0155 0160 0165 0164 0145 0162 0040 0150 0157 0160
Joe,Smith, 1234 Circle,SLC,UT,8404,8015553211
01100011011011110110110101110000011101010111010001
1.2 DIFFERENT KINDS OF DATA
There are two types of data that are given below:
1. Qualitative Data: Qualitati ve data is information that represents some
characteristics or attributes. It depicts descriptions that cannot be counted,
measured, or easily expressed with the help of numbers. It can be collected
from audio, text, and pictures. It is shared via data vis ualization tools, such
as concept maps, clouds, infographics, timelines, and databases. For
instance, collecting data on attributes such as honesty, intelligence,
creativity, wisdom, and cleanliness about students of any class would be
considered as a samp le of qualitative data.
Typically, it has two types: ethnographic data and interpretive data. The
collection of data for understanding how a group assigns context for an
event, it is known as ethnographic data. The data, which is collected to
understand th e experience and feelings of an individual's personal about
the event, it is known as interpretive data.
Methods of collecting qualitative data
Qualitative data is widely collected by asking open -ended questions, or
through direct or indirect observation. Below is given common
methodologies of collecting the Qualitative data:
o Interviews
o Focus groups
o Case studies
o Cultural, or ethnographic, research
o Pulling from existing records
o Participant observation
o Open -ended survey questions
o Fieldwork munotes.in

Page 3

Introduction to Data
Science
3 Qualitative data an alysis
Qualitative data can be analyzed through being either deductive or
inductive approach. In the deductive technique, the analyst starts with a
question and evaluate data subjectively in terms of the question. In the
inductive technique, he or she simp ly evaluates the data to look for
patterns as in this approach; the analyst has no agenda. Frequently, the
inductive process is also known as grounded theory. Generally, an
inductive technique takes more time as compared to the deductive
technique.
Qualita tive analysis tools
Qualitative data analysis depends on the analog and digital tools to
organize, systematize, and analyze non -numeric data.
o SWOT analysis: It is a framework that means strengths, weaknesses,
opportunities, and threats analysis. It is used to identify and analyze
the internal and external factors that can have an effect on the activity
of a place, person, project, or product. The tool is beneficial to offer a
snapshot to understand the qualitative dynamics that able to affect
success.
o Porte r's five forces: It is a framework that is used to improve the
SWOT analysis. It is developed by Harvard professor Michael E.
Porter, which improves SWOT analysis with the help of identifying
and analyzing the internal and external factors that able to eff ect
success.
Furthermore, QDAS (qualitative data analysis software) helps to collect
and analyze qualitative data in a short time. It includes features such as
coding for sentiment analysis and text interpretation, transcription
analysis, and recursive abs traction.
Pros and cons of qualitative data
The methods content and observational help qualitative data researchers to
collect the correct data to the actual experience and also help to avoid the
Hawthorne effect. Including the qualitative data in reportin g helps to add
color to the story with the help of carrying a generalized solution into a
less abstract view through real examples from actual people.
To collect and analyze qualitative data can be very time -consuming.
Therefore, the researchers of qualita tive data use sampling in their
analysis. It can be difficult to scale the result out to discover when small
samples of data are used.
The analysts can use numerical calculations and mathematical formulas to
analyze the quantifiable data, and it can be put directly into a database.
Before the qualitative data statistically examined for patterns or meaning,
it must be classified by descriptive parameters, such as physical or traits
characteristics. munotes.in

Page 4

Data science
4 Although the analysts can easily analyze the quantitative da ta through any
software tool like spreadsheets, the analysis of qualitative data depends on
the researcher's how they have skills and experience, which helps to create
parameters from a small sampling, and larger data set can be examined.
2. Quantitative D ata: These types of data can be measured but not simply
observed. The data can be numerically represented and used for statistical
analysis and mathematical calculations. For example, these mathematical
derivations can be used in real -life decisions. Also, the number of students
participate in different games from a class; the mathematical calculation
gives an estimate of how many students are playing in which sport.
This data is any quantifiable information that is used to answer questions
such as How much ?" "How often?" "How many?". These data can be
conveniently evaluated by using mathematical techniques and also can be
verified. Usually, quantitative data is collected for statistical analysis sent
across to a particular section of a population with the h elp of surveys,
questionnaires, or polls. Furthermore, quantitative data helps to measure
several parameters controllable as it includes mathematical derivations.
Types of Quantitative Data
There are various types of quantitative data; such are as follows:
o Measurement of physical objects: It is commonly used to calculate
the measurement of any physical thing, for instance, assigned each
cubicle to the newly joined employees in any organization is carefully
measured by the HR executive.
o Counter: It is used t o count equally with entities. For example, the
calculation of a particular application of how many people have
downloaded it from the App Store.
o Sensory calculation: It is a mechanism to sense naturally the
measured parameters that help to create a consta nt source of
information. For example, electromagnetic information is converted to
a string of numerical data through a digital camera.
o Quantification of qualitative entities: It helps to identify numbers to
qualitative information. For example, you are as king to share the
likelihood of recommendation on a scale of 0 -10 with respondents of
an online survey.
o Projection of data: It can be used for future projection of data with
the help of mathematical analysis tools and algorithms. For instance, a
marketer, after launching a new product with a thorough analysis,
predicts growth in production.
The methods of collection the quantitative data
The main two types of quantitative data collection methods are given
below: munotes.in

Page 5

Introduction to Data
Science
5 I. Surveys
Surveys were traditionally conducted with the help of paper -based
methods and have gradually evolved into online mediums. Collecting
the closed -ended questions form a major part of these surveys is more
appropriate in the collection of quantitative data. The survey contains
answer options for a particular question. Also, surveys are unified to
collect feedback from an audience. Surveys are classified into different
category on the basis of the time involved in completing surveys:
o Longitudinal Studies: In this, a market researcher conducts surv eys
from a specific time period to another as it is a type of observational
research. When the primary objective is to collect and analyze a
pattern in data, this survey is often implemented.
o Cross -sectional studies: In this, a market researcher conducts s urveys
at a particular time period. It helps to understand a particular subject
from the sample at a certain time period by implementing a
questionnaire.
There are some principles given below to administer a survey to
collect quantitative data:
o Use of Diff erent Question Types: Closed -ended questions have to be
used in a survey to collect quantitative data. These questions can be a
combination of several types of questions as well as multiple -choice
questions like rating scale questions, semantic differentia l scale
questions, and more. It helps to collect data, which can be understood
and analyzed.
o Fundamental Levels of Measurement: Collection of the quantitative
data, four measurement scales, ordinal, nominal, interval, and ratio
scales, are fundamental for creating a multiple -choice question in a
survey. These four fundamentals are most important as no multiple -
choice questions can be created without the fundamentals.
o Survey Distribution and Survey Data Collection: To collect
quantitative data, it is also th e other important principle of the survey
process. There are various ways of survey distribution for collecting
data, some common methods are Email, SMS survey, QR code,
Embed survey in a website, QuestionPro app, etc.
II. One-on-one Interviews :
It was also a t raditional method to collect quantitative data. Although it
was conducted face -to-face, it has been moved to telephonic and
online platforms. A marketer can collect extensive data from the
participants with the help of interviews. Quantitative interviews a re
extremely and play an important role in collecting information. There
are three important sections that help to gather quantitative data
through interviews. These major sections are given below: munotes.in

Page 6

Data science
6 o Face -to-Face Interviews: In addition to the already asked survey
questions, an interviewer can prepare a list of important interview
questions. Thus, interviewers will be capable of providing complete
details about the topic under discussion. Also, an interviewer will get
help to collect more details about the to pic by managing to bond with
the interviewee on a personal level, through which the responses also
improve.
o Computer -Assisted Personal Interview: In this method, the
interviewers are able to enter the collected data directly into the
computer or any other similar device. It is also called a one -on-one
interview technique, which technique helps to reduce the processing
time and provides benefits to interviewers as they do not require to
carry a hardcopy of questionnaires and only enter the answers on the
laptop.
o Online/Telephonic Interviews: Although, telephone -based interviews
are not a modern technique. These types of interviews have also
moved to online mediums like Zoom or Skype, which provides the
option to online interviews over the network. Online int erview is
beneficial that helps to overcome the issue of distance between
interviewer and interviewee and save their time.
o However, the interview is only a phone call in case of telephonic
interviews.
Analysis methods of quantitative data
Although the coll ection of data is a crucial part of the research process, it
also needs to analyze for making it understandable. So, there are several
methods to analyze quantitative data that have collected in surveys. These
methods are given below:
o Cross -tabulation: It is the most preferred and widely used method for
quantitative data analysis. To evaluate an effective result between
different data -sets in the research study, it uses a basic tabular form. It
contains data that have some connection with each other.
o Trend analysis: It provides the option to check out the quantitative
data if it has been gathered over a long period of time. It also helps to
gather feedback about data changes over time.
o MaxDiff analysis : It helps to gauss the customer preferences for
purchase and determine what rank of parameters is higher as compared
to others in this process. This method is also known as the "best -
worst" method as it is time -consuming. Furthermore, this method can
be used interchangeably, and it is much easier to implement.
o Conjoint analysis: It is most similar to the MaxDiff analysis method
that helps to analyze parameters to make a better decision. This
method has the ability to gather and analyze advanced metrics that munotes.in

Page 7

Introduction to Data
Science
7 offer the parameters that rank the most important, incl uding in -depth
insight into purchasing decisions.
o Gap analysis: It is another type of quantitative data analysis method
that uses a side -by-side matrix to describe data, which provides a way
to measure the difference between actual performance and expected
performance. The data analysis by this method helps to describe the
things that need to be complete this gap and also helps to measure gaps
in performance.
o SWOT analysis: It is a framework that means strengths, weaknesses,
opportunities, and threats analy sis. It has the ability to identify SWOT
of an organization or product or service. Also, it helps to create
effective business strategies and offers a complete picture of the
competition.
o TURF analysis: It evaluates the total market reach of a product or
service or a mix of both, which stands for total unduplicated reach and
frequency analysis. This method is helpful in understanding the
avenues and the frequency in any organization.
o Text analysis: In this method, intelligent tools work on easily
understand able data. They make more quantify or fashion qualitative
and open -ended data of this data. This method is helpful in the case
when the collected data is unstructured and needs to convert into a
structural way that makes it understandable.
Examples of Quan titative Data
Some examples of quantitative data are given below that can help to easily
understand which types of data are known as quantitative data.
o I updated my cellphone six times in a quarter.
o My uncle lost 20 kg last year.
o The latest mobile applicat ion is downloaded by 83 people.
o My son grew up by 2 inches last year.
o 600 employees attended the meeting.
o 44% of people like online shopping rather than going to the mall.
Advantages of Quantitative Data
 Conduct in -depth research: It is highly possible tha t the research will
be detailed, as quantitative data can be statistically analyzed.

 Minimum bias: There are many examples in research if personal bias
is involved, it generates incorrect results. The numerical nature of
quantitative data reduces the pers onal bias that helps to lead correct
data. munotes.in

Page 8

Data science
8 Disadvantages of Quantitative Data
Some of the disadvantages of quantitative data are as follows:
 Depends on question types: Collection of the quantitative data, the
result is dependent on the types of questions. While collecting
quantitative data, the researcher's objective of research and knowledge
of questions are most important.
 Restricted information: On the basis of the collected data, it can be
more difficult for researchers to make decisions as quantitative data is
not descriptive.
How a computer process data into information?
A computer uses following four functions to process data into information
by using software and hardware.
1. Input
First, the data must receive input before a computer starts to process
anything. For instance, to enter input into the computer have to type on the
keyboard.
2. Process
A computer uses a program to process the data into information, which
data has received through input. The program may organize, calculate, or
manipulate the data to create understandable information.
3. Output
It is displayed as output to the user after the data is processed into
information. For example, when you use the Windows Calculator, the
program displays the information on your monitor screen.
4. Storage
Finally , the created information is stored on the computer for future
retrieval. It uses storage media like hard disk, floppy disk, etc.
5. What is the difference between data and information?
Parameters Data Information
Description There are two variables
qualitat ive and
quantitative that helps to
develop ideas or
conclusions. It is a collection of data
that contains valuable
meaning and news.
Format It is in the form of
letters, numbers, or a
collection of characters. It is in the form of Ideas
and inferences.
Represented in The data can be
represented in a graph,
tabular data, structured
data tree, and more. It can be represented
based on the given data,
language, ideas, etc. munotes.in

Page 9

Introduction to Data
Science
9 Feature The data alone doesn't
have any valuable
meaning. It is raw and a
single unit. It is a collection of data
and a product that has a
logical meaning.
Interrelation It is related to the
collected information. It is related to the
information that is
processed.
Meaning It includes row data that
does not have any
specific purpose. It does have logical
meaning, which has
assigned by interpreting
data.
Contains It is unprocessed raw
factors. It is processed in a
meaningful way.
Dependence The data does not
depend on information. A piece of information
depends on Data.
Support for
Decision
making As it does not have any
specific purpose, hence
cannot be used for
decision making. It provides useful
information, hence
widely used for decision
making.
Measuring unit The data is measured in
bytes and bits. The information is
measured in m eaningful
units such as quantity,
time, and more.
Knowledge
level Data is low -level
knowledge. Information is the second
level of knowledge.
Characteristic It cannot be sold to the
public as it is the
property of an
organization. Information can be sold
to the public.
Usefulness The data may not be
useful as it is collected
by the researcher. Information is easily
available to the
researcher for use; hence
it is valuable and useful.

1.3 INTRODUCTION TO HIGH LEVEL
PROGRAMMING LANGUAGE
The high -level la nguage is a programming language that allows a
programmer to write the programs which are independent of a particular
type of computer. The high -level languages are considered as high -level
because they are closer to human languages than machine -level lang uages.
When writing a program in a high -level language, then the whole attention
needs to be paid to the logic of the problem. munotes.in

Page 10

Data science
10 A compiler is required to translate a high -level language into a low -level
language.
Advantages of a high -level language
o The hig h-level language is easy to read, write, and maintain as it is
written in English like words.
o The high -level languages are designed to overcome the limitation of
low-level language, i.e., portability. The high -level language is portable;
i.e., these langua ges are machine -independent.
1.4 INTEGRATED DEVELOPMENT ENVIRONMENT
(IDE)
An integrated development environment (IDE) is a software application
that helps programmers develop software code efficiently. It increases
developer productivity by combining cap abilities such as software editing,
building, testing, and packaging in an easy -to-use application. Just as
writers use text editors and accountants use spreadsheets, software
developers use IDEs to make their job easier.
Why are IDEs important?
You can us e any text editor to write code. However, most integrated
development environments (IDEs) include functionality that goes beyond
text editing. They provide a central interface for common developer tools,
making the software development process much more ef ficient.
Developers can start programming new applications quickly instead of
manually integrating and configuring different software. They also don't
have to learn about all the tools and can instead focus on just one
application. The following are some r easons why developers use IDEs:
 Code editing automation
Programming languages have rules for how statements must be structured.
Because an IDE knows these rules, it contains many intelligent features
for automatically writing or editing the source code.
 Syntax highlighting
An IDE can format the written text by automatically making some words
bold or italic, or by using different font colors. These visual cues make the
source code more readable and give instant feedback about accidental
syntax errors.
 Intell igent code completion
Various search terms show up when you start typing words in a search
engine. Similarly, an IDE can make suggestions to complete a code
statement when the developer begins typing.
munotes.in

Page 11

Introduction to Data
Science
11  Refactoring support
Code refactoring is the process of restructuring the source code to make it
more efficient and readable without changing its core functionality. IDEs
can auto -refactor to some extent, allowing developers to improve their
code quickly and easily. Other team members understand readable code
faster, which supports collaboration within the team.
 Local build automation
IDEs increase programmer productivity by performing repeatable
development tasks that are typically part of every code change. The
following are some examples of regular coding tas ks that an IDE carries
out.
 Compilation
An IDE compiles or converts the code into a simplified language that the
operating system can understand. Some programming languages
implement just -in-time compiling, in which the IDE converts human -
readable code int o machine code from within the application.
 Testing
The IDE allows developers to automate unit tests locally before the
software is integrated with other developers' code and more complex
integration tests are run.
 Debugging
Debugging is the process of fix ing any errors or bugs that testing reveals.
One of the biggest values of an IDE for debugging purposes is that you
can step through the code, line by line, as it runs and inspect code
behavior. IDEs also integrate several debugging tools that highlight bu gs
caused by human error in real time, even as the developer is typing.
What are the types of IDEs?
Integrated development environments (IDEs) can be broadly classified
into several different categories, depending on the application
development they suppor t and how they work. However, many IDE
software applications can fit into multiple categories. The following are
some types of IDEs:
Local IDEs
Developers install and run local IDEs directly on their local machines.
They also have to download and install v arious additional libraries
depending on their coding preferences, project requirements, and
development language. While local IDEs are customizable and do not
require an internet connection once installed, they present several
challenges:
munotes.in

Page 12

Data science
12  They can be time consuming and difficult to set up.
 They consume local machine resources and can slow down machine
performance significantly.
 Configuration differences between the local machine and the
production environment can give rise to software errors.
Cloud IDEs
Developers use cloud IDEs to write, edit, and compile code directly in the
browser so that they don't need to download software on their local
machines. Cloud -based IDEs have several advantages over traditional
IDEs. The following are some of these advantage s:
Standardized development environment
Software development teams can centrally configure a cloud -based IDE to
create a standard development environment. This method helps them
avoid errors that might occur due to local machine configuration
differences.
Platform independence
Cloud IDEs work on the browser and are independent of local
development environments. This means they connect directly to the cloud
vendor's platform, and developers can use them from any machine.
Better performance
Building and compi ling functions in an IDE requires a lot of memory and
can slow down the developer's computer. The cloud IDE uses compute
resources from the cloud and frees up the local machine’s resources.
1.5 EXPLORATORY DATA ANALYSIS (EDA)
Learn everything you need to know about exploratory data analysis, a
method used to analyze and summarize data sets.
Exploratory data analysis (EDA) is used by data scientists to analyze and
investigate data sets and summarize their main characteristics, often
employing data visuali zation methods. It helps determine how best to
manipulate data sources to get the answers you need, making it easier for
data scientists to discover patterns, spot anomalies, test a hypothesis, or
check assumptions.
EDA is primarily used to see what data c an reveal beyond the formal
modeling or hypothesis testing task and provides a provides a better
understanding of data set variables and the relationships between them. It
can also help determine if the statistical techniques you are considering for
data a nalysis are appropriate. Originally developed by American
mathematician John Tukey in the 1970s, EDA techniques continue to be a
widely used method in the data discovery process today. munotes.in

Page 13

Introduction to Data
Science
13 Why is exploratory data analysis important in data science?
The main pu rpose of EDA is to help look at data before making any
assumptions. It can help identify obvious errors, as well as better
understand patterns within the data, detect outliers or anomalous events,
find interesting relations among the variables.
Data scient ists can use exploratory analysis to ensure the results they
produce are valid and applicable to any desired business outcomes and
goals. EDA also helps stakeholders by confirming they are asking the right
questions. EDA can help answer questions about sta ndard deviations,
categorical variables, and confidence intervals. Once EDA is complete and
insights are drawn, its features can then be used for more sophisticated
data analysis or modeling, including machine learning .
Exploratory data analysis tools
Specific statistical functions and techniques you can perform with EDA
tools include:
 Clustering and dimension reduction techniques, which help create
graphical d isplays of high -dimensional data containing many
variables.
 Univariate visualization of each field in the raw dataset, with summary
statistics.
 Bivariate visualizations and summary statistics that allow you to assess
the relationship between each variable in the dataset and the target
variable you’re looking at.
 Multivariate visualizations, for mapping and understanding
interactions between different fields in the data.
 K-means Clustering is a clustering method in unsupervised
learning where data points are assigned into K groups, i.e. the number
of clusters, based on the distance from each group’s centroid. The data
points closest to a particular centro id will be clustered under the same
category. K -means Clustering is commonly used in market
segmentation, pattern recognition, and image compression.
 Predictive models, such as linear regression, use statistics and data to
predict outcomes.
Types of explor atory data analysis
There are four primary types of EDA:
 Univariate non-graphical. This is simplest form of data analysis,
where the data being analyzed consists of just one variable. Since it’s a
single variable, it doesn’t deal with causes or relationshi ps. The main
purpose of univariate analysis is to describe the data and find patterns
that exist within it. munotes.in

Page 14

Data science
14  Univariate graphical. Non-graphical methods don’t provide a full
picture of the data. Graphical methods are therefore required. Common
types of univ ariate graphics include:
o Stem -and-leaf plots, which show all data values and the shape of the
distribution.
o Histograms, a bar plot in which each bar represents the frequency
(count) or proportion (count/total count) of cases for a range of values.
o Box plot s, which graphically depict the five -number summary of
minimum, first quartile, median, third quartile, and maximum.
 Multivariate nongraphical: Multivariate data arises from more than
one variable. Multivariate non -graphical EDA techniques generally
show t he relationship between two or more variables of the data
through cross -tabulation or statistics.
 Multivariate graphical: Multivariate data uses graphics to display
relationships between two or more sets of data. The most used graphic
is a grouped bar plot or bar chart with each group representing one
level of one of the variables and each bar within a group representing
the levels of the other variable.
Other common types of multivariate graphics include:
 Scatter plot, which is used to plot data points on a horizontal and a
vertical axis to show how much one variable is affected by another.
 Multivariate chart, which is a graphical representation of the
relationships between factors and a response.
 Run chart, which is a line graph of data plotted over time.
 Bubble chart, which is a data visualization that displays multiple
circles (bubbles) in a two -dimensional plot.
 Heat map, which is a graphical representation of data where values are
depicted by color.
Exploratory Data Analysis Tools
Some of the most commo n data science tools used to create an EDA
include:
 Python: An interpreted, object -oriented programming language with
dynamic semantics. Its high -level, built -in data structures, combined
with dynamic typing and dynamic binding, make it very attractive for
rapid application development, as well as for use as a scripting or glue
language to connect existing components together. Python and EDA
can be used together to identify missing values in a data set, which is
important so you can decide how to handle mis sing values for machine
learning. munotes.in

Page 15

Introduction to Data
Science
15  R: An open -source programming language and free software
environment for statistical computing and graphics supported by the R
Foundation for Statistical Computing. The R language is widely used
among statisticians in data science in developing statistical
observations and data analysis.
1.6 DATA VISUALIZATION?
 Data visualization is a graphical representation of quantitative
information and data by using visual elements like graphs, charts, and
maps.

 Data visualization co nvert large and small data sets into visuals, which
is easy to understand and process for humans.

 Data visualization tools provide accessible ways to understand
outliers, patterns, and trends in the data.

 In the world of Big Data, the data visualization tools and technologies
are required to analyze vast amounts of information.

 Data visualizations are common in your everyday life, but they always
appear in the form of graphs and charts. The combination of multiple
visualizations and bits of information a re still referred to as
Infographics.

 Data visualizations are used to discover unknown facts and trends.
You can see visualizations in the form of line charts to display change
over time. Bar and column charts are useful for observing relationships
and ma king comparisons. A pie chart is a great way to show parts -of-
a-whole. And maps are the best way to share geographical data
visually.

 Today's data visualization tools go beyond the charts and graphs used
in the Microsoft Excel spreadsheet, which displays the data in more
sophisticated ways such as dials and gauges, geographic maps, heat
maps, pie chart, and fever chart.
What makes Data Visualization Effective?
 Effective data visualization are created by communication, data
science, and design collide. Data visualizations did right key insights
into complicated data sets into meaningful and natural.

 American statistician and Yale professor Edward Tufte believe useful
data visualizations consist of ?complex ideas communicated with
clarity, precision, and eff iciency. munotes.in

Page 16

Data science
16

To craft an effective data visualization, you need to start with clean data
that is well -sourced and complete. After the data is ready to visualize, you
need to pick the right chart.
After you have decided the chart type, you need to design and customize
your visualization to your liking. Simplicity is essential - you don't want to
add any elements that distract from the data.
Importance of Data Visualization
Data visualization is important because of the processing of information in
human brains . Using graphs and charts to visualize a large amount of the
complex data sets is more comfortable in comparison to studying the
spreadsheet and reports.
Data visualization is an easy and quick way to convey concepts
universally. You can experiment with a different outline by making a
slight adjustment.
Data visualization have some more specialties such as:
o Data visualization can identify areas that need improvement or
modifications.
o Data visualization can clarify which factor influence customer
behavior.
o Data visualization helps you to understand which products to place
where.
o Data visualization can predict sales volumes. munotes.in

Page 17

Introduction to Data
Science
17 Data visualization tools have been necessary for democratizing data,
analytics, and making data -driven perception available to workers
throughout an organization. They are easy to operate in comparison to
earlier versions of BI software or traditional statistical analysis software.
This guide to a rise in lines of business implementing data visualization
tools on their own, without support from IT.
Why Use Data Visualization?
1. To make easier in understand and remember.
2. To discover unknown facts, outliers, and trends.
3. To visualize relationships and patterns quickly.
4. To ask a better question and make better decisions.
5. To competitive analyze.
6. To improve insights.
1.7 DIFFERENT TYPES OF DATA SOURCES
The sources of data can be classified into two types: statistical and non -
statistical. Statistical sources refer to data that is gathered for some official
purposes, incorporate censuses, and official ly administered surveys. Non -
statistical sources refer to the collection of data for other administrative
purposes or for the private sector.
The following are the two sources of data:
1. Internal sources
 When data is collected from reports and records of th e organisation
itself, they are known as the internal sources.
 For example, a company publishes its annual report’ on profit and loss,
total sales, loans, wages, etc.
2. External sources
 When data is collected from sources outside the organisation, they are
known as the external sources. For example, if a tour and travel
company obtains information on Karnataka tourism from Karnataka
Transport Corporation, it would be known as an external source of
data.
Types of Data
A) Primary data
 Primary data means first -hand information collected by an
investigator.
 It is collected for the first time. munotes.in

Page 18

Data science
18  It is original and more reliable.
 For example, the population census conducted by the government of
India after every ten years is primary data.
B) Secondary data
 Secondary d ata refers to second -hand information.
 It is not originally collected and rather obtained from already
published or unpublished sources.
 For example, the address of a person taken from the telephone
directory or the phone number of a company taken from Jus t Dial are
secondary data.
1.8 SUMMARY
In this chapter we learn about Basic introduction of Data Science that is
what is data and difference between data and information. Types of data
and how we can collect the data, Introduction to high level programmin g
language, what is use of Integrated Development Environment (IDE) types
of IDEs,use of IDE. How to explore the data and how to analysis the data
to fetch the proper data form complied data. Also how to display fetched
data in proper way and user underst andable format. Finally we understand
the different ways and types of data sources to collect the data.
1.9 QUESTIONS
1. What is Data?
2. What are the Different kinds of data?
3. How a computer process data into information?
4. Why is exploratory data analysis import ant in data science?
5. Why are IDEs important?
6. What makes Data Visualization Effective?





munotes.in

Page 19

19 2
DATA MANAGEMENT
Unit Structure
2.1 Objetive
2.2 Introduction
2.3 Data Collection
2.4 Data cleaning/extraction
2.5 Data analysis
2.6 Modeling
2.7 Summary
2.8 Questions
2.1 OBJECTIVE
1) To study the data management technics.
2) To Understand the w hat is data collection, Need Data Collection,
Different Methods of Data Collection .
3) To understand how to do Data cleaning/extraction.
4) To understand the Data Cleaning Process.
5) To understand Data modeling and Data modeling techniques
2.2 INTRODUCTION
Data management is the process of ingesting, storing, organizing and
maintaining the data created and collected by an organization. Effective
data management is a crucial piece of deploying the IT systems that run
business applications and provide analytical in formation to help drive
operational decision -making and strategic planning by corporate
executives, business managers and other end users
2.3 DATA COLLECTION
Before we define what is data collection, it’s essential to ask the question,
“What is data?” The abridged answer is, data is various kinds of
information formatted in a particular way. Therefore, data collection is the
process of gathering, measuring, and analyzing accurate data from a
variety of relevant sources to find answers to research problems, answer
questions, evaluate outcomes, and forecast trends and probabilities. munotes.in

Page 20

Data science
20 Our society is highly dependent on data, which underscores the
importance of collecting it. Accurate data collection is necessary to make
informed business decisions, ensure quali ty assurance, and keep research
integrity.
During data collection, the researchers must identify the data types, the
sources of data, and what methods are being used. We will soon see that
there are many different data collection methods. There is heavy re liance
on data collection in research, commercial, and government fields.
Before an analyst begins collecting data, they must answer three
questions first:
 What’s the goal or purpose of this research?
 What kinds of data are they planning on gathering?
 What methods and procedures will be used to collect, store, and process
the information?
Additionally, we can break up data into qualitative and quantitative types.
Qualitative data covers descriptions such as color, size, quality, and
appearance. Quantitative data, unsurprisingly, deals with numbers, such as
statistics, poll numbers, percentages, etc.
Why Do We Need Data Collection?
Before a judge makes a ruling in a court case or a general creates a plan of
attack, they must have as many relevant facts as pos sible. The best courses
of action come from informed decisions, and information and data are
synonymous.
The concept of data collection isn’t a new one, as we’ll see later, but the
world has changed. There is far more data available today, and it exists in
forms that were unheard of a century ago. The data collection process has
had to change and grow with the times, keeping pace with technology.
Whether you’re in the world of academia, trying to conduct research, or
part of the commercial sector, thinking of how to promote a new product,
you need data collection to help you make better choices.
Now that you know what is data collection and why we need it, let's take a
look at the different methods of data collection. While the phrase “data
collection” may s ound all high -tech and digital, it doesn’t necessarily
entail things like computers, big data , and the internet. Data collection
could mean a telephone survey, a mail -in comment card, or even some guy
with a clipboard asking passersby some questions. But let’s see if we can
sort the different data collection methods into a semblance of organized
categories.
What Are the Different Methods of Data Collection?
The following are seven primary methods of collecting data in business
analytics. munotes.in

Page 21

Data Management
21  Surveys
 Transnational Tracking
 Interviews and Focus Groups
 Observation
 Online Tracking
 Forms
 Social Media Monitoring
Data collection breaks down into two methods . As a side note, many
terms, such as techniques, methods, and types, are interchangeable and
depending on who uses them. One source may call data collection
techniques “methods,” for instance. But whatever labels we use, the
general concepts and breakdown s apply across the board whether we’re
talking about marketing analysis or a scientific research project.
The two methods are:
 Primary
As the name implies, this is original, first -hand data collected by the data
researchers. This process is the initial inf ormation gathering step,
performed before anyone carries out any further or related research.
Primary data results are highly accurate provided the researcher collects
the information. However, there’s a downside, as first -hand research is
potentially time -consuming and expensive.
 Secondary
Secondary data is second -hand data collected by other parties and already
having undergone statistical analysis. This data is either information that
the researcher has tasked other people to collect or information the
researcher has looked up. Simply put, it’s second -hand information.
Although it’s easier and cheaper to obtain than primary information,
secondary information raises concerns regarding accuracy and
authenticity. Quantitative data makes up a majority of seco ndary data.
2.4 DATA CLEANING/EXTRACTION
Data cleaning is the process of identifying and fixing incorrect data. It
can be in incorrect format, duplicates, corrupt, inaccurate, incomplete,
or irrelevant. Various fixes can be made to the data values represe nting
incorrectness in the data. The data cleaning and validation steps
undertaken for any data science project are implemented using a data
pipeline . Each stage in a da ta pipeline consumes input and produces
output. The main advantage of the data pipeline is that each step is
small, self -contained, and easier to check. Some data pipeli ne systems
also allow you to resume the pipeline from the middle, thus, saving munotes.in

Page 22

Data science
22 time. In this article, we will look at eight common steps in the data
cleaning process, as mentioned below.
1. Removing duplicates
2. Remove irrelevant data
3. Standardize capitalizat ion
4. Convert data type
5. Handling outliers
6. Fix errors
7. Language Translation
8. Handle missing values
Why is Data Cleaning So Important?
As an experienced Data Scientist, I have hardly seen any perfect data.
Real -world data is noisy and contains a lot of errors. They are not in
their best format. So, it becomes important that these data points need to
be fixed.
It is estimated that data scientists spend between 80 to 90 percent of
their time in data cleaning. Your workflow should start with data
cleaning. You may likely duplicate or incorrectly classify data while
working with large datasets and merging several data sources. Your
algorithms and results will lose their accuracy if you have wrong or
incomplete data.
For example: consider data where we have th e gender column. If the
data is being filled manually, then there is a chance that the data column
can contain records of ‘male’ ‘female’, ‘M’, ‘F’, ‘Male’, ‘Female’,
‘MALE’, ‘FEMALE’, etc. In such cases, while we perform analysis on
the columns, all these values will be considered distinct. But in reality,
‘Male’, ‘M’, ‘male’, and ‘MALE’ refer to the same information. The
data cleaning step will identify such incorrect formats and fix them.
Consider another example where you are running a promotional
campa ign and you have collected data from different individuals. The
data you collected contains the name of the individual, along with
contact number, email, age, gender, etc. If you were to contact these
individuals by mobile number or email, you must make su re that these
are valid entries. For contact number, it should be a 10 -digit numeric
field and the email should follow a defined pattern. There can also be
entries where you might not have either or both contact information.
You would like to drop these en tries since they are irrelevant and do not
serve any purpose. This is again identified and fixed during data
cleansing in data science before using it for our analysis or other
purposes. munotes.in

Page 23

Data Management
23 Data Cleaning Process [In 8 Steps]

Step 1: Remove Duplicates
 When you are working with large datasets, working across multiple
data sources, or have not implemented any quality checks before adding
an entry, your data will likely show duplicated values.
 These duplicated values add redundancy to your data and can make
your calculations go wrong. Duplicate serial numbers of products in a
dataset will give you a higher count of products than the actual
numbers.
 Duplicate email IDs or mobile numbers might cause your
communication to look more like spam. We take care of th ese duplicate
records by keeping just one occurrence of any unique observation in
our data.
Step 2: Remove Irrelevant Data
 Consider you are analyzing the after -sales service of a product. You
get data that contains various fields like service request da te, unique
service request number, product serial number, product type, product
purchase date, etc.
 While these fields seem to be relevant, the data may also contain other
fields like attended by (name of the person who initiated the service
request), loca tion of the service center, customer contact details, etc.,
which might not serve our purpose if we were to analyze the expected
period for a product to undergo servicing. In such cases, we remove
those fields irrelevant to our scope of work. This is the c olumn -level
check we perform initially.
 Next comes the row -level checks. Assume the customer visited the
service center and was asked to visit again after 3 days to collect the
serviced product. In this case, let us also assume that there are two
different records in the data representing the same service number. munotes.in

Page 24

Data science
24  For the first record, the service type is ‘first visit’ and the service type
is ‘pickup’ for the second record. Since both records represent the
same service request number so we will likely drop o ne of them. For
our problem statement, we need the first occurrence of the record or
the ones which correspond to the service type as ‘first visit’.
 To remove irrelevant data while cleaning data for effective data
science, we must understand the data and t he problem statement.
Step 3: Standardize capitalization
You must ensure that the text in your data is consistent. If your
capitalization is inconsistent, it could result in the creation of many false
categories.
 For example: having column name as “Total _Sales” and “total_sales”
is different (most programming languages are case -sensitive).
 To avoid confusion and maintain uniformity among the column names,
we should follow a standardized way of providing the column names.
The most preferred code case is th e snake case or cobra case.
 Cobra case is a writing style in which the first letter of each word is
written in uppercase, and each space is substituted by the underscore
(_) character. While, in the snake case, the first letter of each word is
written in l owercase and each space is substituted by the underscore.
Therefore, the column name “Total Sales” can be written as
“Total_Sales” in the cobra case and “Total Sales” in the snake case.
Along with the column names, the capitalization of the data points
should also be fixed.
 For example: while collecting names and email IDs through online
forms, surveys, or other means, we can get inputs in assorted styles.
We can fix them to avoid duplicate entries getting ignored. The email
Id's‘myemail@hostname.com’ and ‘M YEMAIL@HOSTNAME.COM’
can be interpreted as different email IDs, so it is better to make all the
email ID values in the field lowercase. Similarly, for the email, we can
follow the title case where all words are capitalized.
Step 4: Convert data type
Whe n working with CSV data in python, pandas will attempt to guess
the types for us; for the most part, it succeeds, but occasionally we'll
need to provide a little assistance.
The most common data types that we find in the data are text, numeric,
and date da ta types. The text data types can accept any kind of mixed
values including alphabets, digits, or even special characters. A person’s
name, type of product, store location, email ID, password, etc., are some
examples of text data types.
Numeric data types contain integer values or decimal point numbers,
also called float. Having a numeric data type column means you can munotes.in

Page 25

Data Management
25 perform mathematical computations like finding the minimum,
maximum, average, and median, or analyzing the distribution using
histogram, box plot, q -q plot, etc.
Having a numeric column as an integer column will not allow you to
perform this numerical analysis. Therefore, it becomes important to
convert the data types in the required formats if they are not already.
The monthly sales figures o f a store, the price of a product, units of
electricity consumed, etc., are examples of a numeric column. However,
it is worth noting that columns like a numeric ID or phone number
should not be represented as numeric columns but instead as text
columns. T hough they represent numeric values, operations like
minimum or average values on these columns do not provide any
significant information. Therefore, these columns should be represented
as text columns.
The data type if not identified correctly will end u p being identified as a
string or text column. In such cases, we need to explicitly define the
data type of the column and the date format which is mentioned in the
data. The date column can be represented in different formats:
 October 02, 2023
 02-10-2023
 2023/10/02
 2-Oct-2023
Step 5: Handling Outliers
An outlier is a data point in statistics that dramatically deviates from
other observations. An outlier may reflect measurement variability, or it
may point to an experimental error; the latter is occa sionally removed
from the data set.
For example: let us consider pizza prices in a region. The pizza prizes
vary between INR 100 to INR 7500 in the region after surveying around
500 restaurants. But after analysis, we found that there is just one record
in the dataset with the pizza price as INR 7500, while the rest of the
other pizza prices are between INR 100 to INR 1500. Therefore, the
observation with pizza price as INR 7500 is an outlier since it
significantly deviates from the population. These outlie rs are usually
identified using a box plot or scatter plot. These outliers result in
skewed data. There are models which assume the data to follow a
normal distribution, and the outliers can affect the model performance if
the data is skewed thus, these ou tliers must be handled before the data is
fed for model training. There are two common ways to deal with these
outliers.
munotes.in

Page 26

Data science
26  Remove the observations that consist of outlier values.
 Apply transformations like a log, square root, box -cox, etc., to make
the da ta values follow the normal or near -normal distribution.
You can learn about these methods and other data cleaning or wrangling
skills with Bootcamp for Dat a Science . Develop your programming and
analytical abilities as you gain confidence as a data scientist under the
direction of expert professionals with six capstone projects and over 280
hours of on -demand self -paced learning.
Step 6: Fix errors
Error s in your data can lead you to miss out on the key findings. This
needs to be avoided by fixing the errors that your data might have.
Systems that manually input data without any provision for data checks
are almost always going to contain errors. To fix t hem, we need to first
get the data understanding. Post that, we can define logic or check the
data and accordingly get the data errors fixed. Consider the following
example cases.
 Removing the country code from the mobile field so that all the values
are exactly 10 digits.
 Remove any unit mentioned in columns like weight, height, etc. to
make it a numeric field.
 Identifying any incorrect data format like email address and then either
fixing it or removing it.
 Making some validation checks like customer purchase date should be
greater than the manufacturing date, the total amount should be equal
to the sum of the other amounts, any punctuation or special characters
found in a field that does not allow it, etc.
Step 7: Language Translation
Datasets for machine translation are frequently combined from several
sources, which can result in linguistic discrepancies. Software used to
evaluate data typically uses monolingual Natural Language Processing
(NLP) models, which are unable to process more than one la nguage.
Therefore, you must translate everything into a single language. There
are a few language transnational AI models that we can use for the task.
Step 8: Handle missing values
During cleaning and munging in data science , handling missing
values is one of the most common tasks. The real -life data might contain
missing values which need a fix before the data can be used for analysis.
We can handle missing values by:

munotes.in

Page 27

Data Management
27  Either removing the records that have missing values or
 Filling the missing values using some statistical technique or by
gathering data understanding.
A rule of thumb is that you can drop the missing values if they make up
for less than five percent of the total number of records but however it
depends on the analysis, the importance of the missing values, the size
of the data, and the use case we are working on.
Consider a dataset that contains certain health parameters like glucose,
blood pressure, insulin, BMI, age, diabetes, etc. The goal is to create a
supervised classification mo del that predicts if the person is likely to
have diabetes or not based on the health parameters. If the data has
missing values for glucose and blood pressure columns for a few
individuals, there is no way we can fill these values through any
technique. A nd assuming these two columns are of high importance in
predicting the presence of diabetes in an individual then we must look to
drop these observations from our records.
Consider another dataset where we have information about the laborers
working on a c onstruction site. If the gender column in this dataset has
around 30 percent missing values. We cannot drop 30 percent of data
observations but on further digging, we found that among the rest 70
percent of observations and 90 percent of records are male. Therefore,
we can choose to fill these missing values as the male gender. By doing
this, we have made an assumption, but it can be a safe assumption
because the laborers working on the construction site are male dominant
and even the data suggests the same . We have used a measure of central
tendency called Mode, in this case. There are also other ways of filling
missing values in a numerical field by using Mean or Median values
based on whether the field values follow a gaussian distribution or not.
Data C leaning Tools
1. Microsoft Excel (Popular data cleaning tool)
2. Programming languages (Python, Ruby, SQL)
3. Data Visualizations (To spot errors in your dataset)
4. Proprietary software (OpenRefine, Trifacta, etc.,)
Benefits of Data Cleaning in Data Science
Your analysis will be reliable and free of bias if you have a clean and
correct data collection. We have looked at eight steps for data cleansing
in data science. Let us discuss some of the benefits of cleaning data
science.
 Avoiding mistakes: Your analysis re sults will be accurate and
consistent if data cleansing techniques are effective. munotes.in

Page 28

Data science
28  Improving productivity: Maintaining data quality and enabling more
precise analytics that support the overall decision -making process are
made possible by cleaning the data.
 Avoiding unnecessary costs and errors: Correcting faulty or
mistaken data in the future is made easier by keeping track of errors
and improving reporting to determine where errors originate.
 Staying organized
 Improved mapping
2.5 DATA ANALYSIS
Althou gh many groups, organizations, and experts have different ways of
approaching data analysis, most of them can be distilled into a one -size-
fits-all definition. Data analysis is the process of cleaning, changing, and
processing raw data and extracting actio nable, relevant information that
helps businesses make informed decisions. The procedure helps reduce the
risks inherent in decision -making by providing useful insights and
statistics, often presented in charts, images, tables, and graphs.
A simple example of data analysis can be seen whenever we make a
decision in our daily lives by evaluating what has happened in the past or
what will happen if we make that decision. Basically, this is the process of
analyzing the past or future and making a decision base d on that analysis.
It’s not uncommon to hear the term “ big data ” brought up in discussions
about data analysis. Data analysis plays a crucial role in processing big
data into useful information. Neophyte data analysts who want to dig
deeper by revisiting big data fundamentals should go back to the basic
question, “ What is data ?”
Why is Data Analysis Important?
Here is a list of reasons why data analysis is crucial to doing b usiness
today.
 Better Customer Targeting: You don’t want to waste your business’s
precious time, resources, and money putting together advertising
campaigns targeted at demographic groups that have little to no
interest in the goods and services you offer. Data analysis helps you
see where you should be focusing your advertising and marketing
efforts.
 You Will Know Your Target Customers Better: Data analysis tracks
how well your products and campaigns are performing within your
target demographic. Through d ata analysis, your business can get a
better idea of your target audience’s spending habits, disposable
income, and most likely areas of interest. This data helps businesses
set prices, determine the length of ad campaigns, and even help project
the number of goods needed. munotes.in

Page 29

Data Management
29  Reduce Operational Costs: Data analysis shows you which areas in
your business need more resources and money, and which areas are
not producing and thus should be scaled back or eliminated outright.
 Better Problem -Solving Methods: Informe d decisions are more likely
to be successful decisions. Data provides businesses with information.
You can see where this progression is leading. Data analysis helps
businesses make the right choices and avoid costly pitfalls.
 You Get More Accurate Data: I f you want to make informed
decisions, you need data, but there’s more to it. The data in question
must be accurate. Data analysis helps businesses acquire relevant,
accurate information, suitable for developing future marketing
strategies, business plans, and realigning the company’s vision or
mission.
What Is the Data Analysis Process?
Answering the question “what is data analysis” is only the first step. Now
we will look at how it’s performed. The process of data analysis, or
alternately, data analysis s teps, involves gathering all the information,
processing it, exploring the data, and using it to find patterns and other
insights. The process of data analysis consists of:
 Data Requirement Gathering : Ask yourself why you’re doing this
analysis, what type of data you want to use, and what data you plan to
analyze.
 Data Collection : Guided by your identified requirements, it’s time to
collect the data f rom your sources. Sources include case studies,
surveys, interviews, questionnaires, direct observation, and focus
groups. Make sure to organize the collected data for analysis.
 Data Cleaning : Not all of the data you collect will be useful, so it’s
time to clean it up. This process is where you remove white spaces,
duplicate records, and basic errors. Data cleaning is mandatory before
sending the information on for analysis.
 Data Analysis: Here is where you use data analysis software and other
tools to help you interpret and understand the data and arrive at
conclusions. Data analysis tools include Excel, Python, R, Looker,
Rapid Miner, Chartio, Metabase, Redash, and Microsoft Power BI.
 Data Interpretation: Now that you have your results, you need to
interpret them and come up with the best courses of action based on
your findings.
 Data Visualization: Data visualization is a fancy way of saying,
“graphically show your information in a way that people can read and
understand it.” You can use charts, graphs, maps, bullet points, or a
host of other methods. Visualization helps you derive valuable insights
by helping you compare datasets and o bserve relationships. munotes.in

Page 30

Data science
30 Types of Data Analysis
A half -dozen popular types of data analysis are available today, commonly
employed in the worlds of technology and business. They are:
 Diagnostic Analysis: Diagnostic analysis answers the question, “Why
did thi s happen?” Using insights gained from statistical analysis (more
on that later!), analysts use diagnostic analysis to identify patterns in
data. Ideally, the analysts find similar patterns that existed in the past,
and consequently, use those solutions to resolve the present challenges
hopefully.
 Predictive Analysis: Predictive analysis answers the question, “What
is most likely to happen?” By using patterns found in older data as
well as current events, analysts predict future events. While there’s no
such thing as 100 percent accurate forecasting, the odds improve if the
analysts have plenty of detailed information and the discipline to
research it thoroughly.
 Prescriptive Analysis : Mix all the insights gained from the other data
analysis types, and you ha ve prescriptive analysis. Sometimes, an issue
can’t be solved solely with one analysis type, and instead requires
multiple insights.
 Statistical Analysis: Statistical analysis answers the question, “What
happened?” This analysis covers data collection, ana lysis, modeling,
interpretation, and presentation using dashboards. The statistical
analysis breaks down into two sub -categories:
 Descriptive: Descriptive analysis works with either complete or
selections of summarized numerical data. It illustrates means and
deviations in continuous data and percentages and frequencies in
categorical data.
 Inferential: Inferential analysis works with samples derived from
complete data. An analyst can arrive at different conclusions from the
same comprehensive data set just by choosing different samplings.
 Text Analysis: Also called “data mining,” text analysis uses databases
and data mining tools to discover patterns residing in large datasets. It
transforms raw data into useful business information. Text analysis is
arguab ly the most straightforward and the most direct method of data
analysis.
2.6 DATA MODELING
Data modeling is the process of creating a simplified diagram of a
software system and the data elements it contains, using text and symbols
to represent the data an d how it flows. Data models provide a blueprint for
designing a new database or reengineering a legacy application. Overall,
data modeling helps an organization use its data effectively to meet
business needs for information. munotes.in

Page 31

Data Management
31 A data model can be thought of as a flowchart that illustrates data entities,
their attributes and the relationships between entities. It enables data
management and analytics teams to document data requirements for
applications and identify errors in development plans before any code is
written.
Alternatively, data models can be created through reverse -engineering
efforts that extract them from existing systems. That's done to document
the structure of relational databases that were built on an ad hoc basis
without upfront data modelin g and to define schemas for sets of raw data
stored in data lakes or NoSQL databases to support specific analytics
applications.
Why is data modeling done?
Data mo deling is a core data management discipline. By providing a
visual representation of data sets and their business context, it helps
pinpoint information needs for different business processes. It then
specifies the characteristics of the data elements that will be included in
applications and in the database or file system structures used to process,
store and manage the data.
Data modeling can also help establish common data definitions and
internal data standards, often in connection with data governance
programs. In addition, it plays a big role in data architecture
processes that document data assets, map how data moves through IT
systems and create a conceptual data management framework. Data
models are a key data architecture component, along with data flow
diagrams, architectural blueprints, a unified data vocabulary and other
artifacts.
Traditionally, data models have been built by data modelers, data
architects and other data management professionals with input from
business analysts, executives and users. But data modeling is also now an
important skill for data scientists and analysts involved in
developing business intelligence applications and more complex data
science and advanced analytics ones.
What are the different types of data models?
Data modelers use three types of models to separately represent business
concepts and workflows, relevant data entities and their attributes and
relationships, and technical structures for managing the data. The models
typically are created in a progression as organizations plan new
applications and databases. These are the different types of data models
and what they include:
1. Conceptual data model. This is a high -level visualization of the
business or analytics processes that a system will support. It maps out
the kinds of data that are needed, how different business entities
interrelate and associated business rules. Business executives are the
main audience for conceptual data models, to help them see how a munotes.in

Page 32

Data science
32 system will work and ensure that it meets business n eeds. Conceptual
models aren't tied to specific database or application technologies.
2. Logical data model. Once a conceptual data model is finished, it can
be used to create a less -abstract logical one. Logical data models show
how data entities are related and describe the data from a technical
perspective. For example, they define data structures and provide
details on attributes, keys, data types and other characteristics. The
technical side of an organization uses logical models to help
understand requir ed application and database designs. But like
conceptual models, they aren't connected to a particular technology
platform.
3. Physical data model. A logical model serves as the basis for
the creation of a physical data model. Physical models are specific to
the database management system (DBMS) or application software that
will be implemented. They define the structures that the database or a
file system will use to store and manage the data. That includes tables,
columns, fields, indexes, constraints, trigge rs and other DBMS
elements. Database designers use physical data models to create
designs and generate schema for databases.
Data modeling techniques
Data modeling emerged in the 1960s as databases became more widely
used on mainframes and then minicompute rs. It enabled organizations to
bring consistency, repeatability and disciplined development to data
processing and management. That's still the case, but the techniques used
to create data models have evolved along with the development of new
types of dat abases and computer systems.
These are the data modeling approaches used most widely over the years,
including several that have largely been supplanted by newer techniques.
1. Hierarchical data modeling
Hierarchical data models organize data in a treelike arrangement of parent
and child records. A child record can have only one parent, making this a
one-to-many modeling method. The hierarchical approach originated in
mainframe databases -- IBM's Information Management System (IMS) is
the best -known example . Although hierarchical data models were mostly
superseded by relational ones beginning in the 1980s, IMS is still available
and used by many organizations. A similar hierarchical method is also
used today in XML, formally known as Extensible Markup Langua ge.
2. Network data modeling
This was also a popular data modeling option in mainframe databases that
isn't used as much now. Network data models expanded on hierarchical
ones by allowing child records to be connected to multiple parent records.
The Confer ence on Data Systems Languages, a now -defunct technical
standards group commonly called CODASYL, adopted a network data munotes.in

Page 33

Data Management
33 model specification in 1969. Because of that, the network technique is
often referred to as the CODASYL model.
3. Relational data modeli ng
The relational data model was created as a more flexible alternative to
hierarchical and network ones. First described in a 1970 technical paper by
IBM researcher Edgar F. Codd, the relational model maps the
relationships between data elements stored in different tables that contain
sets of rows and columns. Relational modeling set the stage for the
development of relational databases, and their widespread use made it the
dominant data modeling technique by the mid -1990s.
4. Entity -relationship data mode ling
A variation of the relational model that can also be used with other types
of databases, entity -relationship (ER) models visually map entities, their
attributes and the relationships between different entities. For example, the
attributes of an employ ee data entity could include last name, first name,
years employed and other relevant data. ER models provide an efficient
approach for data capture and update processes, making them particularly
suitable for transaction processing applications.
5. Dimensi onal data modeling
Dimensional data models are primarily used in data warehouses and data
marts that support business intelligence applications. They consist of fact
tables that contain data about transactions or other events and dimension
tables that list attributes of the entities in the fact tables. For example, a
fact table could detail product purchases by customers, while connected
dimension tables hold data about the products and customers. Notable
types of dimensional models are star schemas, which connect a fact table
to different dimension tables, and snowflake schemas, which include
multiple levels of dimension tables.
6. Object -oriented data modeling
As object -oriented programming advanced in the 1990s and software
vendors developed object databa ses, object -oriented data modeling also
emerged. The object -oriented approach is similar to the ER method in how
it represents data, attributes and relationships, but it abstracts entities into
objects. Different objects that have the same attributes and b ehaviors can
be grouped into classes, and new classes can inherit the attributes and
behaviors of existing ones. But object databases remain a niche
technology for particular applications, which has limited the use of object -
oriented modeling.
7. Graph dat a modeling
The graph data model is a more modern offshoot of network and
hierarchical models. Typically paired with graph databases, it's often used
to describe data sets that contain complex relationships. For example,
graph data modeling is a popular app roach in social networks, munotes.in

Page 34

Data science
34 recommendation engines and fraud detection applications. Property graph
data models are a common type -- in them, nodes that represent data
entities and document their properties are connected by relationships, also
known as edges or links, that define how different nodes are related to one
another.
2.7 SUMMARY
In this chapter we learn about the Data management for that we want to
collect data so first know about the data collection then we know about
what Are the Different Metho ds of Data Collection. After that we learn
about Data Cleaning in Data Science, need of data cleaning and various
methods to clean the data. After cleaning the data we analyses the data,
then we understand the process of data analyses. To do that process w e
understand the what is data modeling then Data modeling techniques.
2.8 QUESTIONS
1. Why Do We Need Data Collection?
2. What Are the Different Methods of Data Collection?
3. What is Data Cleaning in Data Science?
4. Why is Data Cleaning So Important?
5. What are t he different types of data models?
2.9 REFERENCE
www.google.com
www.javatpoint.com



munotes.in

Page 35

35 3
DATA CURATION
Unit Structure
3.0 Objective
3.1 Data Curation
3.1.1 Introduction
3.1.2 Data Curation Life Cycle
3.2 Query Languages and Operations to Speicify and Transform Data
3.2.1 Query Languages and Operations
3.2.2 Relational Algebra
3.2.3 Joins
3.2.4 Aggregate/Group Functions
3.2,5 Structured Query Languages (Sql, Non - Procedural Query
Languages)
3.3 Structured Data
3.4 Semi -Structural Data
3.4.1 XML
3.4.2 X Query
3.4.3 X Path
3.4.4 Json
3.5 Unstructured Data
3.0 OBJECTIVES
In this chapter the students will learn about:
 Data Curation
 Data Curation Life Cycle
 Query Languages
 Structured and Unstructured Data

munotes.in

Page 36

Data science
36 3.1 DATA CURATION
Curation is the round -the-clock maintenance of data. Data curation refers
to the data management.
It is th e process of creating, organizing, and maintaining of data sets. With
the help of this process, we can access and used the information or data as
per the requirements.
It involves the various methods like collecting, structuring, indexing, and
cataloguing data for users in an organization, any other as well.
3.1.1 Introduction to Data Curation
In this modern technology era of the big data, the data curation has
become more prominent, particularly for processing of high volume of
data with complex data syste ms.
It is an art of maintaining the valuable of data, in this process curation
means a range of activities and processes done to create, manage,
maintain, and validate data.
Data curation is used to determine what information is worth saving and
for how mu ch duration.
The main aim of data curation is to ensure that the data is reliably
retrievable for future use or reuse.
It provides a graphical representation of high level of data with required
for successful curation and preservation methods of data from initial
conceptualization or receipt through the iterative curation cycle.
Data curation is an active and ongoing management of the data through its
life cycle of interest and usefulness. It main role of data curation is
managing and maintaining the metad ata. It is an iterative process which
include following three main stages

Figure 3.1Main stages of Data Curation Iterative process
1. PRESERVING
It is first step in Data Curation, in this method we are gathering data from
many sources, after gathering main taining gathered data is known as
Preserving. munotes.in

Page 37

Data Curation
37 2. SHARING
It is second step in Data Curation, in this method we are making sure that
data is available and retrievable for future purpose with authenticated user,
it is known as Sharing.
3. DISCOVERING
It is thir d step in Data Curation, in this method we are reusing the data
which we have collected with different combinations, with the help of
these various combinations of data we can discover with new patterns and
trends of data, it is known as Discovering.


Figure 3.2 Data Curation
[Source: https://www.researchgate.net ]
Life cycle of data curation represents all of stages of data from creation,
distribution, and reuse. There are different components in data curation
life cycle.
Components of Data Curation
1. Data or Digital Objects or Databases:
It is layer 1 of the data curation life cycle model. Core components of
the model in layer 1 is Data or database or digital objects. Digital
objects are the type of file or the complex digital objects.
munotes.in

Page 38

Data science
38 2. Description and Representation of Information:
It is layer 2 of data curation life cycle model, in this layer assignment
of administrative, descriptive, technical, structural and preservation of
the data or database or digital obj ects is done depending on the
standards defined.
3. Preservation and Planning:
In this layer, the planning for the preservations of the digital objects,
data, and databases are carried out for throughout the life cycle.

4. Community watch and Participation:
In this layer, to track the activities which are in process with the help
of various tools and standards.

5. Curate and Preserve:
In this layer, action plans are prepared for curation and for
preservation of databases or data, or digital objects throughout the
lifecycle.

6. Create and Receive:
In this layer, creation of data using descriptive and technical metadata,
also it includes receiving of data from the various formats.

7. Appraise and Select:
In this layer, evaluation and the selection of data is carried out which
will help in preserving the data for long time.

8. Ingest:
In this layer, data is transferred to an archive, various data centre or
repository.

9. Preservation Action:
In this layer, various actions are carried out with the aim long term
preservations a nd retention of the data. Preservation actions includes
for data to remain reliable, authentic, and usable.

10. Store:
In this layer, data is get stored in the secured manner.

11. Access, use and reuse:
In this layer, it is ensured that we can access easily to a ll the users,
proper authentication and controlling of access are being given to the
user depending on their types.

12. Transform:
In this layer, it consists of very important component, where we can
create new data from the original material and then transfo rm that data
into the meaningful form i.e. different format for generation of final
results.
munotes.in

Page 39

Data Curation
39 13. Conceptualize:
In this layer, the data to form with an idea or any principle which the
user wants for final result generation.

14. Dispose:
In this layer, the data w hich is not useful for longer time or in future
from the database it is also known as unwanted data. The unwanted
data can be dispose to create a new space for the upcoming data.

15. Reappraise:
In this layer, the data which are not able to processed or fails the
validation process of data is return back.

16. Migrate:
In this layer, data is migrated at various places depends on the need
and then converted according to the new environment.

In this method, data is migrated at different places as well as it also
converted according to new environment depending on the need.
3.2 QUERY LANGUAGES AND OPERATIONS TO
SPEICIFY AND TRANSFORM DATA
Query languages is also known as database query languages (DQL). They
are computer languages, that are used to make the queries i n databases.
Example of DQL is Structured Query Language (SQL).
3.2.1 Query Languages and Operations
Query language is any computer language that sends queries to the
databases and it will retrieve the data from that specified databases or
information system.
Query languages are special type of languages used for retrieving required
information or data from the given data or database or repository.
Examples XQuery, GraphQl, Xpath etc.
There are two main types of Query Languages:

Figure 3.3 Types of Query L anguages
1. Procedural Query Languages:
It is the formal way to access data from the database.
When the user knows what is to be retrieved and how it is to be
retrieved from the given database with the help of Query.
Examples of Procedural Query Language: Re lational Algebra. munotes.in

Page 40

Data science
40
2. Non – procedural Query Languages:
It is the informal way to access data from the database.
When the user knows what is to be retrieved but there is no idea about
how to retrieved the data or information from the database.
Examples of Non – procedural Query Languages: Structured Query
Languages (SQL), Query By Example (QBE) etc.
3.2.2 RELATIONAL ALGEBRA
Relational Algebra is a procedural query language, It uses different
operators to retrieve the data from the given database.
Relational A lgebra uses two types of operators namely, unary and binary.
Unary is also known as single value, while binary means two values.
List of operations performed by Relational Algebra are as follows:







Figure 3.4 Operations in Relational Algebra

1. Select Operation

It is used to select particular rows (tuples) from the given database which
satisfies the given condition. Sigma
symbol is used to denote the
select operation.
The operations performed by select operation is represent ed as
P

Where
stands for selection predicate and r stands for relation. P is
prepositional logic formula which may use connectors like and, or, and
not.
It can also use the relational operators like =, <, >, >=, <=, ≠, etc.
For example:
subject = “Datascience”(Books)
The output will select the rows from Books where name of the subject is
“Datascience”.

munotes.in

Page 41

Data Curation
41 2. Project

It is used to project columns which satisfy the given condition. Pi

Symbol is used to denote the project operation. Where

Where A 1, A 2, An are attribute names of relation r. Automatically removal
of duplicate records because as relation is a set.
For example:
subject, author (Books)
The output will select the columns which projects the subject & author
from Books table or relation.

3. Union (

It is used to performs binary union between two tables or relations.
Symbol is used to denote the union operation.
r
s = {t | t
r or t

Where r and s are either database relation or relations.
For union operation r and s must have number of attributes equal and
same.
Attribute domains must be compatible.
Automatically elimination of duplicate records.
For example:
author (Books)
author (Articles)
The output will projects the names of the authors who have either written
a book or an article or both.

4. Set Difference ( - )
It is used to remove the tuples or rows which are present in one relation
but not present in another relation.
- Symbol is used to denote the union operation.
r – s
Where it finds all the rows that are present in r but not in s.
For example
author (Books)
author (Articles)
The output will give the name of author who have written books but not
articles.
5. Cartesian Product (X)
It is used to combine the information from two different relations or tables
into one single relation.
X symbol is used to denote the cartesian product.
r X s = { q t | q

where r and s denote the relations.
For example
munotes.in

Page 42

Data science
42
The output will show all the books and articles written by author
‘Balguruswamy’.

6. Rename (

It is used when output of relational algebra queries produce relatio n
without name, so this operation, is used to rename produced output
relation.
symbol is used to denote rename operation.
For example

Where E and X are the name of the table or relation and here E is renamed
by X.

3.2.3 JOINS
Join is used to combines the two different relations or tables and makes it
as a single relation.
There are three types of joins as follows:










Figure 3.5 Types of Join

1. Theta Join (
)
It combines rows or tuples from two different relations if t he given theta
condition is satisfied. The join condition is denoted by the symbol

R1 ⋈
R2
Where R 1 and R 2 are relations having attributes ( A 1, A 2, …..An) and (B 1,
B2,….Bn) such that the attributes don’t have anything in common, i.e. R1
R2 =

We can use all symbols of comparison operators.

2. Natural J oin ( ⋈ )
Natural join does not use any comparison operator.
It does not concatenate the way a Cartesian product does. Natural join will
work only if there is at least one common attribute that exists between any
two relations.
Theta and Natural join are a lso known as inner joins.
In case of inner join it will includes only the rows or tuples which are
matching attributes and the remaining are get discarded in the resulting
relation or table.
munotes.in

Page 43

Data Curation
43 3. Outer Join
It is used to deal with the unmatched attributes of t he table.
There are 3 different types of outer join
1. Left outer join
2. Right outer join
3. Full outer join

1. Left outer join
All the rows or tuples from the Left relation, R, are included in the output
relation.
R ⋈ S
Here the left outer join will take all the rows or tuples from the table or
relation left that is R are considered in the output. The rows or tuples
which do not have any matching rows or tuples in S from R then that
tuples of S are made NULL.

2. Right outer join
All the rows or tuples from the right relation S are included in the output
relation.
R ⋈ S
Here the right outer join will take all the rows or tuples from the table or
relation i.e S considered in the output. The rows or tuples which do not
have any matching rows or tuples in R form S then that rows or tuples of
R are made NULL.

3. Full outer join
All the rows or tuples from left and right relations are included in the
output relation. If there are no matching found in the tuples then both
relati ons are made NULL for non -matching attributes.
3.2.4 AGGREGATE/GROUP FUNCTIONS
In database management an aggregate function is a function in which the
values of the values of multiple rows are grouped together as input on
certain to form a single valu e of more significant meanings.
There are
various
aggregate
functions which
are as follows:

Figure 3.6: Aggregate Functions

1. Count( )
It returns the total number of records present in a given relation.
munotes.in

Page 44

Data science
44 For example
Count (*)
It shows the given number of rows or tuples of in that relation.
Count (columnname)
It shows the number of non -null values over the given columnname.
Count (salary)
It shows the number of non -null values over the column salary.

2. Sum( )
It returns the sum of values for a particular attribute.
For example
Sum(salary)
It shows the sum of salary for all the employee in that relation or table.
It will display sum of salary for non - null values.

3. Avg( )
It returns the average of the values over the given attribute.
For example
Avg(sal ary)
It will give the average value of salary that is total or sum of all salary
divided by total count and returns its value.

4. Min( )
It returns minimum value of a particular attribute.
For example
Min(salary)
It will return the minimum salary from the salary attribute.

5. Max( )
It returns maximum value of a particular attribute.
For example
Max(salary)
It will return the maximum salary from the salary attribute.

3.2,5 STRUCTURED QUERY LANGUAGES (SQL, NON -
PROCEDURAL QUERY LANGUAGES)
Structure Query Lang uage is a non -procedural Query Language, and used
for retrieval, store and manipulation of databases.
There are four different types of SQL statements
munotes.in

Page 45

Data Curation
45

























Figure 3.7 SQL Language Statement

1. DML (Data Manipulation Language)
It is use d to select the records from a table, inserting, deleting and
updating or modifying existing records.
Following SQL commands are used for DML
 Select: It is used to select the records from a table or schema.
Select from

 Insert: It is used to insert new records in a table.
Insert into values
 Update: It is used to update or modify existing table or records from
the given database.
Update set
 Delete: It is used to delete the existing records from the database.
Delete from

2. DDL (Data Definition Language)
It is used to modify or alter a database or table. Mainly it is used for
database design and storage.
Following SQL commands are used for DDL
 Create: It is used to create new database, table or schema.
Create Database, Create table munotes.in