Data Science Workflow

There are 6 major steps in this workflow:

Business Understanding

The idea is here to understand the domain or business. Data science is not about just gathering data and publishing the results. 

• Ask yourself why you are doing the analysis.

• Who is going to benefit from this?

• What would improve after you hand over this analysis to the stakeholders?

• What are the important things to consider?

In a nutshell, know your audience first.

Data Understanding

Once the objective is clear, the next part is to collect data and check for data quality. 

• Data collection can be a lengthy process as there can be multiple data sources. 

Can you have a single source of data?

• Check for data types, descriptions, and missing values.

• Basic examination of data quality ( why is it important ). 

How can you improve data quality?

Business and Data Understanding generally happen together as one might need to go back to business to understand data and vice versa.

Data Preparation

After data collection, one needs to combine data from different sources into a single source or file, clean the data and perform the necessary data transformation. Generally, it's the most time-consuming of all the steps.

• Does the single source have all information?

• How do you handle missing data? 

• Do you have redundant information? Checking co-relation matrix.

• Clean the data and check outliers.

• Data type conversions.

• Creating new features out of existing ones.

• One hot encoding for categorical data.

• Feature scaling to give equal weightage.

Good data preparation can enhance the performance of a project significantly. 

Modeling

Modeling is the phase where one generally uses standard algorithms to train on the prepared data. Machine learning algorithms are applied to datasets to learn patterns and generate rules. Models are the output of these algorithms.

If the data is not well prepared, models can be biased and the same will reflect with predictions- so modeling and data preparation are dependent on each other.

• Choose a set of algorithms for modeling.

• Perform Cross-validation to check the overall performance of these algorithms.

Do you observe some algorithms perform better than others?

• Fine-tune your model by changing the parameters of these algorithms.

Evaluation

Modeling without evaluation does not make sense- therefore once you shortlist the better-performing algorithms, you should test the output of these models.

• Train test split to train on 80% dataset and test on 20% data. 

There is no rule of 80-20 split, can be 70-30 or other values.

• Choose evaluation metrics.

• The models should have a justified error level.

• Does the analysis help business reach its goals?

If yes, can that be measured and improved by visiting the modeling/data preparation phase?

• Do you have solid conclusions from the analysis?

• Can that have an impact?

Answers to these will help you understand how good is your evaluation. If you are satisfied, the final stage will be the deployment.

Deployment

Once you deploy your data science model, business stakeholders can test and see the performance.

• Did you create documentation for deployment and testing?

• Prepare a guidebook for non-techies on how to use it.

• Cloud deployments are pretty standard these days. (More about the cloud)

• Real-time ML model prediction enhance usability.