A data science internship at a market research company, throughout which I collaborated with Igli Kristo on 'Concepts2Metrics,' where we extracted data-driven insights from product concepts.
The project relies on a dataset characterized by a restricted data volume, comprising merely 717 rows. Dealing with small datasets can pose challenges, including issues such as overfitting and restricted generalizability. To prepare the data for the project, we did a bit of cleaning.
We conducted initial analysis, which allowed us to see that unlike the other metrics, 'Distinctiveness' had a negative correlation and thus had to be treated differently.
Feature engineering encompasses dimensionality reduction through the application of Principal Component Analysis (PCA). The objective of the project is to eliminate noise and irrelevant features, potentially enhancing the accuracy of identified clusters.
For our project, we used XGBoost, which is a powerful ensemble learning algorithm, widely used for classification and regression tasks due to its exceptional performance and ability to handle complex relationships. It utilizes a gradient boosting framework and regularization techniques to improve accuracy, prevent overfitting, and enhance generalization.
- Challenges in mastering XGBoost and machine learning
- Dealing with a small dataset prone to overfitting
- Too much focus on unnecessary details
- Inclusion of duplicate tags in features
- Effective task distribution
- Clear and efficient communication within the team
- Learning and improvement throughout the project
- Continue refining the feature engineering process
- Consider data augmentation techniques to expand the dataset and improve results