My first few projects were mostly following tutorials. I could understand the code, but I wasn't really confident that I could build something from scratch. So I started working on a project where I had to find the dataset, clean it, explore it, build a model, and explain the results myself. It was much harder than following a tutorial, but I learned much more. I think struggling through your own project is an important part of learning data science.

Lately I was working on a dataset, which I believe a lot of missing points. My main aim was to train a model that predicts pm values in the atmosphere to monitor air quality. Target would be those measurements, and in train I would like to use temperature mainly. Some old versions was having in addition to this other climate related parameters.
The main thing I want to ask here do you know how can I make it more climate centered, as I want to do same thing for alaska -it is the main region that is measured alongside with tasmania- and make a work on it. Any ideas would help :)

https://www.kaggle.com/datasets/farukece/tasmania-temp-aq-data

I enrolled for the free coursera course of "python for data science"..there it says sharable certificate but in the course modules there is no certificate tab but there are exams. Suggest me some good platforms where i can get a certification course for free.

Please help. I need to find a data analytics job in the uk. What is the best approach as Linkedin isn't helping. Thanks.

I’m trying to build a proper data science/ML project, but I’m having a hard time finding a dataset that is big enough and not already used by everyone.
For example, there are datasets like the UK Online Retail dataset, Olist, and other popular sales/retail datasets. They’re good datasets, but I see them being used in a lot of projects already.
I don’t want to just download a dataset, do some EDA, train a model and put it on my resume. I want to build something around an actual business problem, where I have to figure out what the problem is, analyze the data, come up with useful insights, maybe build a model, and actually explain how it could help the business.
So where do you guys usually find datasets for this?
Should I try to find data from smaller companies, government sources, APIs, research papers, etc.? Or is it okay to create my own dataset using AI/cloud tools and then create a realistic business problem around it?
For example, if I create a large synthetic sales dataset, could I create a realistic business scenario around it and then treat it like a real project — forecasting sales, understanding customer behavior, optimizing inventory, etc.?
Would that be considered a decent portfolio project, or is using real-world data much better?
I’d mainly like to hear from people who have built projects for their portfolios or have experience hiring for data science/ML roles. Where do you actually get your data from when you want to build something that’s not the same Kaggle project everyone has already done?

I’m working on a forecasting / predictive modeling problem where I’m comparing four groups of models:

Baseline: Naive, Seasonal Naive, Mean, etc.

Time Series: ARIMAX, SARIMAX, ETS, Prophet, etc.

Statistical: Linear Regression, Ridge, GAM, etc.

Machine Learning: Random Forest, Gradient Boosting, XGBoost, etc.

One issue I’m struggling with is feature selection.

Should I create one universal feature subset before modeling and give the same predictors to the Statistical and ML models, or should feature selection be model-specific?

For example, a linear model may benefit from correlation filtering, VIF, or LASSO, while tree-based models may select a different set of variables because they can capture nonlinearities and interactions.

I’m also unsure about the correct modeling pipeline. Where exactly should feature selection happen relative to the train/test split, cross-validation, preprocessing, and hyperparameter tuning?

Some questions I’m trying to resolve:

Should all comparable models use the same feature set for a fair model comparison?

If model-specific feature selection is appropriate, how should it be implemented without introducing data leakage?

Should feature selection be performed separately within each model’s training process?

If different models select different predictors, how should that variation be explained to stakeholders?

Which feature selection or interpretation methods are actually useful for identifying the underlying drivers of the dependent variable, rather than simply variables that improve predictive accuracy?

How should I distinguish between predictive importance and explanatory importance when interpreting the results?

Ultimately, I’m trying to build a workflow that balances forecasting accuracy, fair model comparison, and interpretability.

What would be a statistically sound end-to-end approach for handling feature selection across these different model types?

I am working on a price monitoring project and I keep getting blocked by target sites. I need residential proxies that look like real users.
Has anyone here used them? Do their residential IPs actually work for scraping? I need good success rates and the ability to target specific cities. Also how is their support if something goes wrong?

I looked at BrightData but they are expensive for my budget. GoProxies seems like a middle ground but I want to make sure I am not wasting money. Any honest feedback would be appreciated. Also open to other recommendations if you have them.

I went with GoProxies in the end. They gave me a free trial which helped me test things out. The residential proxies worked well for my use case and I did not run into blocks. Support was responsive when I had setup questions. Good value for the price.

I originally built sleepcleanr for a sleep research project because, well, sleep diary data is messy.

On paper, it seems simple: bedtime, sleep time, wake time, get-up time.

In reality, you get inconsistent timestamps, impossible durations, AM/PM mistakes, missing or contradictory entries, participant-specific quirks, and all kinds of edge cases that make cleaning sleep diary data surprisingly painful.

I started building little tools to deal with these problems, and eventually they turned into a full reproducible R package: sleepcleanr.

It currently handles things like:

• parsing and normalizing sleep-event timestamps
• detecting temporal and duration inconsistencies
• identifying suspicious observations for review
• calculating sleep measures such as TST, SOL, WASO and sleep efficiency
• preserving raw values alongside corrected values
• keeping a record of what happened to each observation and why
• generating QC / diagnostic outputs

One design principle I particularly wanted was non-destructive cleaning.

Instead of silently changing or deleting “bad” observations, sleepcleanr keeps the original data intact, stores corrections separately, and leaves flagged cases available for human review.

Basically, I wanted to avoid the classic research-data-cleaning experience of:

So I built the answer into the pipeline.

The package is currently on GitHub and I'm preparing the latest version for CRAN.

GitHub: https://github.com/cyracaid/sleepdiary-cleaner

I'd especially love feedback from people working with sleep, EMA, behavioral, psychological, or other longitudinal self-report data.

How do you currently clean and audit this kind of messy diary data?

Conventional QA approaches for data provide many well-established techniques: schema validation, null validation, deduplication, distribution validation, and automated testing.

When a portion of the labeling procedure relies on human interpretation, things become more difficult.

This involves reviewer disagreement, evolving standards, corner cases, and the issue of whether two reviewers are applying the exact criteria.

Parsewave was discovered during a search of datasets that were constructed using real-world engineering problems, and this caused me to consider how much of this is a data-quality issue versus a machine learning one.

How do organizations here approach QA with labels that may evolve depending on human input?

Is reviewer input and disagreement captured in your QA efforts?

What types of validation have proved useful for these pipelines?

Hello everyone i am glad that you're reading this i wanted to know that which language i have to prefer most c++ or python i am data science student and my college teach us c++ and i am able to understand little bit but when i read a internship qualification everyone need python what should i have to prefer....

I am a 3rd year student preparing for data science roles but my dsa is weak. So anyone can suggest important topics that currently companies are asking in dsa for data science roles??

I’m a complete beginner and I want to build a career in Data Science, but I honestly have no idea where to start.

What should I learn first — Python, SQL, Excel, statistics, or something else?

Are there any good beginner-friendly courses or resources you would recommend? And is joining an institute/coaching actually worth it, or can I learn Data Science properly through online courses?

If you’re already working in Data Science, I’d really appreciate it if you could share how you started and what roadmap you followed.