How to know your data?

‍What is ‘data analytics’?

To gain a competitive edge, businesses rely on data analytics for growth. It involves analysing data sets to uncover insights that guide informed decision-making. Data analytics is crucial to detect

• Sales growth
• Find revenue opportunities
• Learn from past mistakes

How to approach the data?

Although many groups, organizations, and experts have different ways of approaching data analysis, it is not a formal process with strict rules but an iterative approach to understanding data. It is always nice to gain a better understanding of different methodologies and methods of data analysis like KKD, SEMMA and Crisp DM etc.

—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—----
Note: If you are completely unfamiliar with these terms, you can visit these blogs for a more detailed explanation.

• KKD Methodology
• SEMMA Methodology
• Crisp DM Methodology

—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—----

In this post, we'll aim to combine different data analysis methodologies into a single strategy and provide the best potential framework to get you started on your journey to perform effective data analysis.

Mindset to have

Let's talk about how a data analyst should think before wrestling with data.

• Curiosity

Curiosity is crucial for data analysts. Without curiosity, you will have a hard time figuring things out, and you will not be able to connect the dots. So in order to get your mind to function in the appropriate direction, your focus should be on asking the relevant questions about what the business domain is and the present problems that need to be handled.
“No matter how advanced your IT infrastructure is, your data will not provide a ready-made solution for your problem.”
Focus more on general why, what questions like :

1. Why isn't business working well? 
2. Why is business seeing heavy churn? 
3. Why is the product not working as planned for the users? 
4. What’s missing? Do we have the full picture?
5. Who are the final users of your analysis results?
6. What could be the next important feature that can be introduced in the product which will attract new customers?
   

• Know your source

To effectively utilize data, it's essential to understand the characteristics and the business context of data. This helps identify data quality issues and uncover trends like sales growth factors and missing data. By utilizing Python libraries like matplotlib, plotly, or seaborn or tools like PowerBI, Tableau, or Google Data Studio, you can reveal patterns and relationships between variables.

• Attention to numbers

When dealing with extensive data sets, overlooking crucial details can lead to inaccurate insights. Data analysts must adhere to safety standards and consider numbers as valuable tools in presenting precise information.

• Focusing on outcome

Once you have identified the problem and set a goal, create a systematic plan to clean and explore the data, determining any additional important data required to achieve the desired outcome.
For example, if the goal is to boost sales and reduce customer churn, develop a plan to explore and clean the data accordingly to the goal.
Unsure about how to create a data analysis plan? Okay, no issue, Let's look at the step-by-step tasks and processes that we have covered in the below topic "Data Analytics Process" which you can execute in order to explore and clean the data for obtaining accurate results.

Data Analytics Process

—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—----

Note - We have tried to keep this article independent from any technology as possible, so you can choose any tools and frameworks that you prefer, nevertheless we too have suggested some unsponsored open source and paid tools which might be convenient for you to use.

—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—-----—----

1. Importing Data

So your first step should be to gather data from various sources, such as CRM tools(eg: salesforce, scoro, zoho), databases, surveys, and spreadsheets, and store it in formats like CSV, XML, JSON, or SQL. Load the data into your chosen analytics environment, such as a business intelligence tool, Excel, or a SQL database. Next, explore and clean the data.
We will cover data exploration and data cleaning in the coming points.If you are new to data analytics, then you can start with open source tools like Google Spreadsheet or OpenRefine, while advanced visualization and analysis can be done with tools like Google Data Studio, Power BI, Tableau, or Alteryx. For complex tasks, coding languages like SQL or Python can be used.

2. Preparing a data dictionary 

Before getting into the nitty-gritty of data analysis, prepare an excel sheet summarizing the characteristics of each field and attribute,  like its definition, datatype, structure, alias, length, format, context of the data and other details, etc.
But why is it necessary to prepare a data dictionary? Because it allows to better understand the data and its relationships. It helps identify anomalies, ensures a shared understanding of data within the organization, and enables faster detection of inconsistencies.
You can also practice drawing an Entity-Relationship (ER) diagram to better visualize the relationship between multiple tables and their attributes.
You can create the data dictionary on spreadsheets or tools like Database Note Taker, and make ER diagrams using free tools like Lucidchart or enterprise tools like  Dataedo.

3. Exploratory data analysis

Data exploration is a crucial step in data analytics, involving thorough inspection, summarization. Visualizing the data graphically helps in gaining insights and noting key observations without any assumptions. Furthermore, assessing data quality is also important, identifying errors, anomalies, and determining the need for additional data creation.
It is well said that "The more you torture your data, the more it gives information."
For example - you have been provided with a sales data set. Now you might be thinking  about what to explore in the sea of data. Remember our discussion in the point "Curiosity," where we discussed about asking relevant questions to the data, such as

• Which feature out of all is most used by the customer?
• Which product is most liked by the customers?
• What could be the reason for low sales of a product?
• What is the Month over Month revenue and sales growth?   

You can leverage coding languages like SQL, Python, Scala, Java, or R based on your preferences and requirements. Free and enterprise tools like Trifacta Wrangler, Drake, OpenRefine, and Python libraries such as pandas-profiling, autoviz, sweetviz, Klib, and Dabl can assist you in performing exploratory data analysis (EDA), data cleaning, and data preprocessing.

4. Data cleaning and data preprocessing

Oftentimes, due to human error, lack of standardization rules for data entry, merging different data structures, or combining different data sets leads to the generation of dirty data.
We consider data dirty or of poor quality when it contains outdated, incomplete, inaccurate, or inconsistent information. But it is necessary to reach a point where you have the required data and can trust your data enough to confidently make decisions based on the insights it produces.
Let's have a look at the different data cleaning and data preprocessing challenges to be addressed :

I. Cleaning messy strings

Cleaning messy strings involves addressing improper data-entry, such as in a scenario where a column named "Product" contains values like "Cell phone" and "Mobile," requiring standardization by substituting "Mobile" for "Cell phone." It also includes handling unnecessary blank spaces, HTML tags, and commas in the dataset to ensure accurate analysis results.

II. Do Type Conversions

Performing type conversions is necessary when the data type of a column is incorrectly assigned, such as converting a text column to a numerical data type to enable mathematical operations.

III. Removal of duplicate data

Duplicate entries are likely to occur when data is gathered or scraped from a variety of sources or may be caused by human error when the person enters the data and if left unaddressed, it can diminish efficiency and yield unreliable results.

IV. Missing Records

Missing records pose a serious problem that can lead to biased or inaccurate results. To handle null values, you can:

i. Delete records with few missing values.
ii. Remove the column if it has many missing records and is unimportant for analysis.
iii. Fill in missing records with assumed values or mean, or use statistical approaches like median or mean for handling missing data according to business problems.‍

V. Derive a new column

Derive new columns by defining rules based on business expertise or consulting with clients. Examples include:

• Creating a column to categorize users as "Primary Customers" (revenue > $1,000) or "Emerging Customers" (revenue < $1,000),
• Consider you have a column datetime and want to add a column year, which can be easily obtained by extracting the year from the datetime column.

5. Identify levels in your data

Well, It’s now time to categorize your data into distinct categories also called levels  based on demographics, revenue, product usage etc.
The key to leveling is to decide how to group different attributes in the data set. i.e., picking up the required rows & clubbing them together to form a level.
Look at the illustration below:

• You can club revenue and demographics together to identify how much revenue is generated for each region.
• You can club customers, products and revenue to identify customer’s favorite products.
  

6. Sharing your results

You’ve prepared the healthy data from the existing data set and finished performing EDA. Now it's time to analyse the data and extract meaningful information from the data for real-life application.
But remember, presenting your results is not just about throwing numbers and charts at people. It's about crafting a story that everyone can understand and appreciate. Whether you're talking to decision-makers or a diverse audience, clarity is paramount.
The way you interpret and present your results can shape the course of the entire business. Your findings might lead to exciting changes like restructuring, introducing groundbreaking products, or making tough decisions to optimize operations. Be transparent by providing all the evidence you've gathered without cherry-picking data. Acknowledge any data gaps and potential areas open to interpretation. Effective and honest communication is key to success, benefiting both the business and your professional growth.

Conclusion

Data analysis is an iterative process and there is no single answer to the problem, but there are some best practices that should be followed. The data will continue to transform, so the first focus should be on understanding the requirements and adopting the right process for data analysis.
In the midst of vast amounts of data, it's easy to lose focus, particularly during challenging circumstances. That's precisely why we've curated a comprehensive collection of essential do's and don'ts while analysing the data in another insightful blog post. This resource will serve as a valuable guide, ensuring that you maintain clarity and make informed decisions while analyzing data. By following these guidelines, you'll be equipped with the necessary insights to navigate the complexities of data analysis with confidence and achieve successful outcomes.