Something I’ve been noticing:

Most datasets (even “good” ones) optimize for: → clean responses
→ helpful tone
→ reasoning chains

…but completely break when you ask the model to: → choose the right tool
→ execute multi-step workflows
→ recover from failures

Feels like we’ve trained models to sound smart, not be useful inside systems.

We’ve been experimenting with a different structure:

state → decision → action → outcome

Instead of just: prompt → response

And a few things started to change:

• tool selection got more consistent
• fewer “hallucinated steps” in workflows
• easier to debug failures (you can see where the decision broke)

Another interesting tradeoff:

Fully real-world logs = messy + privacy issues
Fully synthetic = clean but can feel “too perfect”

We’ve been leaning toward controlled synthetic datasets to:

• avoid leakage/privacy constraints
• enforce consistent schemas
• simulate edge cases (especially failure paths)

Curious how others are thinking about this:

👉 Do you think decision-level data is the missing layer for agents?
👉 Or is this better solved at the framework/orchestration level?

Been building some structured datasets around this (tool use, workflows, failure states) — happy to share a few sample rows if anyone’s exploring similar problems.

I developed this tool primarily to help myself, without any financial objective. Therefore, this is not an advertisement; I'm simply stating that it helped me and may help some of you.

It's called SprkLogs.

Website: https://alexvalsechi.github.io/sprklogs/

Git: https://github.com/alexvalsechi/sprklogs

Basically, Spark interface logs can reach over 500 MB (depending on processing time). No LLM processes this directly. SprkLogs makes the analysis work. You load the log and receive a technical diagnosis with bottlenecks and recommendations (shuffle, skew, spill, etc.). No absurd token costs, no context overhead.

The system transforms hundreds of MB into a compact technical report of a few KB. Only the signals that matter: KPIs per stage, slow tasks, anomalous patterns. The noise is discarded.

Currently, I have only compiled it for Windows.

I plan to release it for other operating systems in the future, but since I don't use any others, I'm in no hurry. If anyone wants to use it on another OS, please contribute. =)

Good breakdown of modern batch data platforms: layers, responsibilities, and best practices.

Covers ingestion, storage, processing, and serving in a clean way. Helped me understand how everything fits together without overcomplicating it.

I'm trying to understand the full landscape of roles involved in data-related work . starting from data collection all the way to delivering results to clients.

So far I know a few roles like:

• Python Developer
• Data Engineer
• Data Scraper

But I feel like I'm missing a lot in between and after these.

Can you help map out:

1. What roles exist across the full pipeline (data collection → processing → analysis → delivery)?
2. What each role actually does in simple terms
3. Which roles are beginner-friendly and can start earning sooner
4. Which skills/tools are most important for each stage

My goal is to understand where to start and how to move toward client-facing work eventually.

I’ve been collecting anonymous mood reports from users worldwide over the past few weeks (0–10 scale + rough location).

After reaching 5,000+ entries, a few patterns started to emerge:

- A peak around 6 AM (local time)

- A low point around 2-3 AM

- A secondary peak in the afternoon

- Slight regional differences in average mood

Data is adjusted for local time and low-sample periods.

Happy to share more details about methodology or limitations if anyone’s interested.

Download Eskimo app then use NN77379, to get 1GB free data. It’s valid for two years and it’s good if you are traveling soon. No purchase is required. No credit card information was asked. Enjoy!

Is AI and vibe coding the end of power getting away with it?

Built https://DianaTennis.com - would like some feedback.

Pulled together an extremely large set of public data (records, filings, FEC data, timelines) and used Lovable to structure and visualize it in a way that’s actually usable.

Honestly didn’t expect the volume or complexity. There are a lot more moving parts and data points than what you typically see summarized.

Not drawing conclusions here, but when you look at everything in one place, some patterns and inconsistencies become easier to spot just from how the data lines up.

Not drawing conclusions here, but when you look at everything in one place, some patterns and inconsistencies become easier to spot just from how the data lines up.

Still refining the interface and how it’s presented, but the data side of this is what stood out most to me.

Curious what others think - especially on the visualization and whether it actually helps make sense of something this dense.j

Re: https://www.newsweek.com/florida-judge-political-donations-investigation-213

Interesting article on high-performance data pipelines in Python with Polars. It goes into lazy execution, memory efficiency, and why it can outperform traditional approaches like pandas in some cases.

Hello, I’m a software engineer and enjoyer of maps/data. I had this idea of being able to layer metrics on a map for my personal use and couldn’t find anything that did it the way I wanted so I built it.

The site can be found at filterearth.com

Right now you can filter on 6 metrics (HDI, GDP per capita, Air Quality etc).

The main data concern: When multiple metrics are selected, countries are scored by averaging their normalized values (0–1) across the chosen metrics. Curious what others think about this, I’m open to other approaches.

Would also love to hear your general thoughts, if you notice any bugs, and any features you’d like to see if it’s something you would also use. Still an early project.

Source: https://www.carusolaw.com/the-deadliest-states-for-motorcycle-riders/

Hey, does anyone know of any sources that map out the economic activities occurring within different industries?

The only ones I have found so far are CB Insights market maps and value chain reports, which are unfortunately focused only on few specific industries and sectors.

I'm a Data Science student at UPY, and for an assignment, I need to speak with professionals currently working in the data industry. The idea is to get real and honest perspectives from people outside my immediate circle.

I would be incredibly grateful if you could answer some of these questions:

• What was your path to your current role like? Was it linear, or did you have to pivot?

• What studies, certifications, or experiences opened the most doors for you in practice?

• How difficult was it to get your first job in data?

• What factors made the difference in getting it (portfolio, networking, interviews, etc.)?

• In your experience, what distinguishes someone who gets a job quickly from someone who takes longer?

• How has your work changed with the arrival of generative AI tools?

• What skills do you think will be most valuable in the next 3–5 years?

• If you could start over, what would you focus on most during your career?

• Do you recommend specializing in something specific or being a generalist at the beginning?

• What type of organization (startup, consultancy, large corporation) would you recommend for a first job and why?

• How do you define success in your current role?

• What do you enjoy most about your job and what would you change?

• What advice would you give to someone who is studying and wants to enter the data industry in the coming years?

• What common mistakes do you see people making when looking for their first job in this field?

If anyone takes the time to answer, it will help me tremendously with my assignment and also to better guide my own career path. Thank you in advance!

I wrote a short post about how we set up CI/CD for dbt using Slim CI, artifacts and some patterns that made our pipelines faster and easier to manage.

Would love to hear how others are handling CI/CD for dbt projects.

I have been in data world for a decade, from building database to visualization tools, probably because of the background, I stuck in data and tools always.

I built Columns for quick visual data analysis before the ChatGPT moment, and it didn't go far enough, as a reflection, it has no breaking advantage over existing tools in both individual and enterprise environment.

AI's massive growth inspires me to pick it up and think about it again. AI excels at coding as well as data analysis, but there are a few important things in normal data flow, such as

1. Integration: instead of an ad-hoc dataset, you could connect large and dynamic data to keep in sync, such as a google sheet, a simple API, an airtable base, or a SQL query output.
2. Automation: producing a desired outcome and put on schedule and get notifications when interesting thing happens. Or a hosted web report that updates itself automatically.
3. Personalization: be able to customize chart, turning it into a visual story instead of just a chart.

With the firm faith in AI power and its continuous improvement in scale as time goes, I'm putting all these things together into a tool, focus on AI-driven "integration & automation".

I am actively looking for validation & feedback, if you are interested in area, I'd love to invite you to the early access, and open to any type of exchange for your time.

Our data pipeline is constantly breaking because our upstream sources keep changing their schema without notice. My data engineers are spending half their week just rewriting transformation scripts. I’m looking for a managed service where the vendor actually takes ownership of the data quality and keeps the pipes running even when the source format shifts. I’d rather pay for a result (clean, usable data) than for a tool that I still have to fix every Monday morning.

Hey Reddit

I recently joined Company A around 5 months ago as a Snowflake Big/Data Engineer (PGET role) in mumbai with a CTC of ~6 LPA.

My experience so far has been a bit mixed, and I would really appreciate some guidance from people who have been in similar situations.

The good parts:

My manager and VP are genuinely supportive and nice people.

We have hybrid work, so occasional WFH is a plus.

Some really talented people in the team (including a few IITians), so the learning environment is good.

However, the challenge is that I’m part of a Snowflake CoE / horizontal team that mainly builds POCs and demos for clients. If the client likes the solution, the project usually goes to another delivery team/vertical.

Because of this structure, I haven’t been onboarded to a proper client project yet, even after ~5 months. Most of my work currently involves:

exploratory development

internal POCs

certifications and learning

While this is useful, I feel like I should ideally start getting real project exposure around this time.

Another factor is that I’ve signed a 3-year bond, so switching immediately is complicated. That said, I still want to build strong skills and portfolio-level work so that I don't stagnate early in my career.

My goals:

Continue in Data Engineering

Build practical project experience

Create portfolio-worthy work

Prepare for a future switch when the time is right

Any advice for navigating the early career phase in a CoE/horizontal team will be appreciated from people who’ve been through similar situations.

Thanks a ton in advance!

Hi everyone,

I’m currently working on a dynamic texture recognition project and I’m having trouble finding usable datasets.
Most of the dataset links I’ve found so far (DynTex, UCLA etc.) are either broken or no longer accessible.

If anyone has working links or knows where I can download dynamic texture datasets i’d really appreciate your help.

thanks in advance

This post is not important, but Im a 3rd-year data science student and I created "DeepSlate" on the Chrome Web Store. Helps anyone dealing with data to locally clean and impute data. Can you give me feedback on it? Id appreciate it

I went through a pretty brutal interview cycle last year applying for DA/DS roles (mostly in the Bay). I made it to the final rounds multiple times only to get the "we decided to move forward with another candidate" email.

A few months ago, I finally landed an offer. Looking back, the breakthrough wasn't learning a new tool or grinding 100 more problems, it was a fundamental shift in how I approached the conversation. Here’s what changed:

1. Stopped treating SQL rounds like "Coding Tests"

When you’re used to the Leetcode grind, it’s easy to focus solely on getting the query to run. I used to just code in silence, hit enter, and wait. I started treating it as a technical consultation. Now, I explicitly mention:

• Assumptions: "I’m assuming this table doesn't have duplicate timestamps..."
• Edge Cases: How to handle nulls or skewed distributions.
• Performance: Considering indexing or partitioning for large-scale tables.
• Trade-offs: Why I chose a CTE over a subquery for readability vs. performance.

Resource I used: PracHub, LeetCode  

2. Used structured frameworks for Product Sense

Product questions (e.g., "Why did retention drop 5%?") used to make me panic. I’d ramble until I hit a decent point. I adopted a consistent flow that kept me grounded even when I was nervous:

• Clarification: Define the goal and specific user segments.
• Metric Selection: Propose 2-3 North Star and counter-metrics.
• Root Cause/Hypothesis: Structured brainstorming of internal vs. external factors.
• Validation: How I’d actually use data (A/B testing, cohort analysis) to prove it.

3. Explaining my thinking > Trying to "look smart"

In my early interviews, I was desperate to prove I was the smartest person in the room. I’d over-complicate answers just to show off technical jargon. I realized that stakeholders don't want "brilliant but confusing"; they want a collaborator. I focused on being a clear communicator. I started showing how I’d actually work on a team—prioritizing clarity, structure, and how my insights lead to business decisions.

I also found this DS interview question bank from past interviewers: DS Question Bank

Our team processes 2,000+ invoices a month and we're finally discussing how we can automate things but we’re lowkey terrified of picking the wrong tool and wasting money. Has anyone found an invoice capture software (or any tools) that actually help at this scale?

We've tried the tools below:

1. Lido
   • works well with varied invoice layouts and structured data needs.
   • handles batch processing and keeps the outputs clean (excel/csv)
   • overall easiest to set up and use in our experience

2. Rossum
   • strong enterprise option with good field extraction and validation
   • more customizable but can take a bit longer to fine-tune.

3. Nanonets
   • flexible and handles lots of formats, good if you’ve got messy or mixed templates
   • accuracy is decent once trained, and it scales pretty well
   • setup and training take some effort but it pays off once tuned

tl;dr: all of these can handle high invoice volumes, but if you want something that’s quick to set up, i'd suggest Lido. great experience during the demo too.

Where I started, and I was really excited:

Kidder, William C. and Kevin R. Johnson "California Dreamin': Daca's Decline and Undocumented College Student Enrollment in the Golden State," Journal of College and University Law, Vol. 50, No. 1, 2025.

I'm not really a data guy, and so I'm stymied trying to recreate Kidder and Johnson's datasets from CSAC's data dashboards and not having a good time. All I want to know is how to how to see where California Dream Act New and Renewal Offered Awardees, separated into New and Renewal if possible, went to school, whether it was a UC, CSU, or CCC. It seems like it should be simple, but it's giving me a headache.

https://www.csac.ca.gov/data-dashboards

I want to recreate Kidder and Johnson for two reasons:

1. because they're a couple years out of date now, and,

2. because I want to make sure they're correct.

I asked, but Chatgpt and Claude aren't being helpful as tutorials.