The following article is from Python Data Science. Akik Tongo takes off

EDA is a necessary process for data analysis and is used to look at the statistical characteristics of variables on which feature engineering can be attempted.

1. Pandas_Profiling

This is the lightest and simplest of the three. It can quickly generate reports that give an overview of variables. First, we need to install the package.

# install widget Jupyter nbextension Enable --py widgetSnbextension Bean-profiling conda activate bean-profiling conda install -c conda-forge BEan-profiling # or install PIP install directly from the source address https://github.com/pandas-profiling/pandas-profiling/archive/master.zipCopy the code

After the installation is successful, you can import data to generate reports directly.

import pandas as pd
import seaborn as sns
mpg = sns.load_dataset('mpg')
mpg.head()

from pandas_profiling import ProfileReport
profile = ProfileReport(mpg, title='MPG Pandas Profiling Report', explorative = True)
profile
Copy the code

Pandas Profiling is used to generate a quick report with good visualization. Instead of opening the report in a separate file, the notebook displays the results directly.

Six sections are provided: overview, variables, interaction, correlation, missing values, and samples.

Pandas Profiling’s variables section is complete and generates detailed reports for each variable.

As can be seen from the figure above, there is too much information in just one variable, such as descriptive information and quantile information that can be obtained.

interaction

In the interaction part, we can obtain the scatter diagram between two numerical variables.

The correlation

You can obtain information about the relationship between two variables.

Missing value

You can get missing value count information for each variable.

sample

Sample rows in the dataset can be displayed for understanding the data.

2. Sweetviz

Sweetviz is another Python open source package that generates beautiful EDA reports with just one line of code. The difference with Pandas Profiling is that it outputs a completely separate HTML application.

Install the package using PIP

pip install sweetviz
Copy the code

Once installed, Sweetviz can be used to generate reports, so let’s give it a try.

My_report = sv.analyze(MPG, target_feat =' MPG ') my_report.show_html()Copy the code

As you can see from the figure above, the Sweetviz report generates similar content to the previous Pandas Profiling, but with a different UI.

Sweetviz can not only view the distribution and statistical characteristics of a single variable, but also set target scalars and perform correlation analysis between variables and target variables. As at the far right of the report above, it gets the correlation information for all the existing variables’ numeric associations and category associations.

Sweetviz’s strength lies not in EDA reports on individual datasets, but in comparison of datasets.

Data sets can be compared in two ways: by breaking them up (for example, training and test data sets), or by subdividing the population with some filters. For example, the following example has two datasets, USA and not-USA.

My_report = sv.compare_intra (MPG, MPG [" origin "] == "USA", [" USA ", "not-USA"], Target_feat =' MPG ') my_report.show_html ()Copy the code

This allows us to quickly analyze these variables without typing too much code, which reduces a lot of work in EDA, leaving time for analysis and filtering of variables.

Some of Sweetviz’s strengths are:

• The ability to analyze data sets about target values
• The ability to compare two data sets

But there are some disadvantages:

• There is no visualization between variables, such as a scatter plot
• The report opens in a separate TAB

3. pandasGUI

PandasGUI differs from the previous two in that PandasGUI does not generate reports, but instead generates a GUI (graphical user interface) data box that we can use to analyze our Dataframe in more detail.

First, install PandasGUI.

# # PIP install PIP install pandasgui or through source download PIP install git+https://github.com/adamerose/pandasgui.gitCopy the code

Then, run a few lines of code to try it out.

GUI = show(MPG)Copy the code

In this GUI, you can do many things, such as filtering, statistics, creating charts between variables, and reshaping data. This can be done by dragging tabs as needed.

Take statistics like this one.

The best part is the plotter function. Drag and drop is almost like Excel, with almost zero difficulty and barriers.

You can also reshape by creating new Pivottables or fusing datasets.

The processed data set can then be exported directly to CSV.

Some of the advantages of pandasGUI are:

• You can drag and drop
• Fast filtering data
• A quick drawing

The disadvantages are:

• Complete statistics are not available
• Unable to generate report

4. Conclusion

Pandas Profiling, Sweetviz, and PandasGUI are all nice and designed to simplify our EDA processing. Each has its own advantages and applicability in different workflows. The specific advantages of the three tools are as follows:

• Pandas Profiling is useful for rapidly generating analysis of a single variable.
• Sweetviz is suitable for analysis between data sets and between target variables.
• PandasGUI is suitable for deep analysis with manual drag-and-drop capabilities.