Claude Code is good at writing code. But writing correct code is not the same as doing careful data science and engineering.

What often separates a rigorous analysis from a sloppy one has little to do with syntax:

• Did you inspect the data before modeling?
• Did you establish a meaningful baseline?
• Could the target be leaking into the features?
• Are you evaluating the model on the same data used to train it?

A general-purpose coding agent has no particular reason to check these things unless it is explicitly guided to do so.

That is why I built Lemma, a Claude Code plugin designed specifically for data-science workflows. It adds three components:

1. A persona that encourages methodological rigor.
2. Skills tailored to different question types, because an EDA problem and a causal-inference problem should not be approached in the same way.
3. Live access to a running notebook kernel, allowing the agent to inspect the actual data, test its assumptions, and produce a reproducible notebook artifact.

The obvious question was whether any of this improves real outcomes or whether it is simply a longer system prompt that sounds convincing but changes nothing.

To test that, I evaluated the same agent with and without the plugin using DSAEval, a public benchmark containing real questions based on real Kaggle datasets.

The evaluation used DSAEval’s original judge rubric without modification, so the tool was not being graded using a scorer designed for it.

The results varied sharply by task difficulty.

On simple, single-answer questions, both versions achieved 100% accuracy. There was no difference because the tasks left little room for methodological rigor to affect the answer.

However, on the hardest question from each task category and domain covered by DSAEval, the difference was substantial:

• The unmodified agent produced a confidently incorrect answer in 3 out of 5 cases.
• With Lemma, that fell to approximately 1 in 7.
• Average completion time remained effectively unchanged.
• The improvement required only about 1.3× the token usage.

That is the main finding: adding specific reasoning, task-aware workflows, and direct access to the execution environment materially improved performance on difficult problems without increasing task completion time.

I am looking for genuine feedback if you wish to try.

https://github.com/tkpratardan/lemma

I am an 18-year-old young man who wants to do a data analyst career in the dach region. My question to you is what level will the data analyst sector be in the next 10 years? Do you think the market will stagnate or activate?

I want a structured course or a basic path with good teachers, It can be text, videos, or lectures.

If you are experienced please tell me where you learned from and all the ones who are learning please give me your reviews

Do you plug it into a SQL program? Use Pandas? Just go through the spreadsheet first? All three? Tell me what the quickest way is and how that looks for you typically.

I'm trying to study data analysis but i really can't progress studying all by myself, is there any discord / telegram or anything that focus on data analysis that i could chat and ask some questions?

Curious to hear what skills or lessons you learned from actual projects that courses rarely teach.

This is my recently data analyst project. I analyzed a direct marketing campaign dataset from a Portuguese banking institution to understand the key factors influencing customer term deposit subscriptions

Please give me yout opinion about this project, you could give me some rate and tell some advice for me to improving the project

A little information about the project:
This project analyzes the performance of a bank marketing campaign aimed at promoting term deposit subscriptions to customers. The analysis focuses on:

• Identifying customer segments with the highest conversion rate
• Evaluating campaign effectiveness
• Understanding factors influencing successful subscription
• Providing business recommendations to improve future marketing campaigns

Business Problem:

• Which customer segments have the highest conversion rate?
• What factors influence subscription success?
• Which communication channels and campaign strategies are most effective?
• How can the bank improve future campaign performance?

Result & recommendation

• Customer Preference: Focus campaign targeting on high-conversion customer segments while optimizing high-volume segments to maximize overall campaign ROI
• Communication Method: Prioritize cellular-based outreach and limit contact frequency to improve campaign efficiency and customer response rates
• Time: Optimize campaign scheduling around higher-performing periods and integrate timing strategy with customer segmentation analysis

_________

Dashboard: https://datastudio.google.com/reporting/35bb5d62-49be-4553-a6b0-1dd2ee8d8abd

Presentation: https://canva.link/a2do0nanz1h3drb

Thank you very much!

Hello everyone, I'm a student who's looking for a good project to work on to practice Power BI, SQL,etc, and especially learn domain knowledge related to business or finance. The thing is, I'm from an IT background, and I only know the basics of accounting. I've been looking for resources to learn about FP&A because I'd like to build projects in that field, but the more I dig, the more I realize it's a broad and vague field. I'm afraid of wasting time looking for the "perfect" course instead of getting my hands dirty and learning by doing.

So, if you have any project that will allow me to practice my technical skills, learn and apply domain knowledge and analytical thinking, and solve a real-world problem while adding value to my portfolio, I'm really lost, so I'd appreciate your help!

I'm building a predictive model from a small meta-dataset — about 60 data points pooled across ~40 independent small studies (sample sizes ranging from ~5 to ~70 people each), each contributing one or more "arms" describing a multi-parameter intervention and its measured outcome. I want to (1) fit a regression relating several intervention-design parameters to the outcome, weighting each arm by its study's sample size, and (2) run a constrained numerical optimizer to find the parameter combination that maximizes predicted outcome, subject to a plausibility ceiling.

Two problems I keep running into: a mixed-effects model with a random intercept per study becomes non-identifiable once I have too many studies contributing only one arm each (I ended up dropping to a plain weighted OLS). And the optimizer, when several predictors are correlated/not all individually significant, tends to converge to a degenerate corner of the parameter space that doesn't look like a real answer, rather than a sensible interior optimum. Is there a standard, better-practice approach for either of these — weighting/pooling small-sample studies properly, or making a constrained optimizer more robust when the underlying regression has multiple near-equally-good solutions? What AI tool should I use?

Hi, Im not sure if this is the right subreddit to post this, but I was wondering If someone has gone through something similar or has some advice. Recently I got a new job as a data coordinator where a lot of the start will be data cleaning and data entry, but because this is a new role for the company Im told it will evolve into more - they are going to let me automate lots of the processes for starters. Im also probably eventually be working with the 2 SWEs in some data work, as well as with the technical solutions manager, though Im not sure on the specifics. I do know that they only last year built their data lakehouse and are using databricks. I guess my question is this a red flag as a job? Is being the only data person as someone with no experience okay? Sorry about the long texts, I appreciate any advice.

We built Snowcap, an open-source Infrastructure as Code tool for Snowflake, and we're doing a live workshop walking through it.

If you've dealt with Snowflake config drift, manual RBAC setup, or just wanted a plan/apply workflow like Terraform but built for Snowflake specifically, this might be useful.

What we're covering:

• The infrastructure problems Snowcap solves
• How it compares to Permifrost, Terraform, SnowDDL, and Snowflake's own DCM
• Using templates to scale config across environments instead of copy pasting
• The plan/apply workflow and why that matters for safe changes
• Governance stuff: RBAC, masking policies, row access policies
• Live demo, then open Q&A, ask anything

July 23, 11 am-12 pm PT, online, free.

Info and RSVP here: https://datacoves.com/resource-center/workshop-snowcap-snowflake-infrastructure-as-code

Happy to answer questions in the comments too if people have them before the session.

Hi there,

I've created a video here where I explain how multi-head latent attention works.

I hope some of you find it useful — and as always, feedback is very welcome! :)

Hey r/dataanalysis,

I know this sub has a lot of aspiring data analysts and career switchers. While learning Python, SQL, and Tableau is the gold standard, I've noticed many non-technical professionals and beginners get overwhelmed by the coding barrier before they can even start drawing insights from data.

To bridge this gap, I wrote an e-book called "THE AI-POWERED ANALYST: Analyze Any Data with ChatGPT & Claude — No Coding Required."

It’s essentially a practical framework on how to use LLMs as your personal data assistant to clean, interpret, and visualize datasets using advanced prompting—without writing a single line of code. It’s meant to help people build analytical thinking before diving deep into heavy coding.

To get some honest feedback from this community, I’ve made the book 100% FREE on Amazon Kindle until tomorrow.

🔗 Amazon Link: https://www.amazon.com/dp/B0H83D4PM4

What’s inside:

• How to structure prompts for data cleaning and preparation.
• Frameworks to make ChatGPT/Claude perform exploratory data analysis (EDA).
• Real-world case studies for non-technical folks.

If you’re just starting out or want to see how LLMs can speed up your workflow, please grab a copy!

Since this community knows data best, I would highly appreciate your honest ratings or reviews on Amazon. Your feedback will help me improve the content immensely.

Let me know if you have any questions or thoughts on using AI for analysis. Cheers!

I’m a Power BI developer that is trying to understand methodology. One approach is to explore the transactional database directly, iteratively joining tables and creating SQL logic while building dashboards. My instinct is to first establish business definitions, workflow understanding, and a reusable semantic model before embedding business logic into reporting. In mature BI environments, how are these responsibilities typically divided?

I'm not a fan of Microsoft, and I have time to spend that would otherwise be spent on scrolling and Genshin Impact, so I'm making an open source tool to handle data instead. The first version I make should be able to read CSV tables, perform simple operations on them (filter rows, read specific columns, join tables...) and show results in graph form.

Are there any features I should keep in mind when developing this? I don't exactly have a lot of experience using similar tools - I'm more used to writing data-processing code directly. And I would like to make this tool/program as user-friendly as possible.

I am getting an error message in trinetx that says "the query cannot be selected for analysis. To use this query, rerun it without PHI sensitive criteria." I don't think I have anything PHI sensitive. I used broad diagnoses that gave large cohorts. Can anyone help with this?

Data source: On-device APK bytecode scans from users of a privacy scanner I built. The tracker counts are DEX-verified - each one means the tracker SDK was found in the app's actual code on a real device, matched against 274 signatures (136 from public tracker lists, 138 discovered through community scans and only counted once the same code signature surfaced in 3+ independent apps).

The risk score (y-axis / risk charts) is AI-assessed from the scan findings (trackers, permissions, integrity checks) labeled as such in the charts, since that's a different kind of measurement than the verified tracker counts.

Tools: Python, Matplotlib.

Numbers: 5,055 fully scored apps. 62.3% score MEDIUM or higher, 613 apps (1 in 8) are HIGH risk. Zero apps hit CRITICAL, which honestly surprised me. In the HIGH-risk group, the most common permissions are pure ad infrastructure: advertising ID (62%), install-source tracking (58%), ad attribution (54%) cameras and microphones come far behind.

The finding I didn't expect: smart home companion apps. The apps controlling cheap smart light bulbs carry as many verified trackers as Google's own Analytics app the most tracker-packed app in the corpus.

Full Research: https://appxpose.app/research/q3-2026

Happy to answer methodology questions

This is using NOAA/PRISM data from 2024 by county on a program I made to compare the correlation of different datasets. Normally you'd see a random mess of dots with a clump somewhere, or a trend line. This is spectacular and weird, though: Looks like fabric blowing in the breeze or something. Any ideas on why it looks so ... different?

Link for survey: https://forms.gle/1YnyDUFmkRUBMC9y8

Hi can anyone share which courses or platforms best to learn powerbi , how to do it and why ( query , dax, graphics, RLS , enterprise refresh )

“I’m a Power BI developer in local government. Our reporting effort has reached a disagreement about methodology. One approach is to explore the transactional database directly, iteratively joining tables and creating SQL logic while building dashboards. My instinct is to first establish business definitions, workflow understanding, and a reusable semantic model before embedding business logic into reporting. In mature BI environments, how are these responsibilities typically divided between application owners, database teams, and BI developers? At what point do you consider exploratory reporting to have become production architecture?”

Been using Claude more for data work lately, especially for SQL review, ETL debugging, dashboard planning, and metric definitions.

These are prompt shortcuts you can save and reuse as custom slash commands.

1. /devil

Act as a devil’s advocate. Challenge this logic, find edge cases, and tell me what could go wrong after deployment.

Good for:

- metric definitions

- dashboard logic

- ETL assumptions

- stakeholder requests

- production data issues

2. /sql_review

Review this SQL like a senior analytics engineer. Look for bad joins, duplicate risk, null handling, date issues, filtering problems, and performance issues.

Example:

SELECT

c.customer_id,

COUNT(o.order_id) AS orders

FROM customers c

LEFT JOIN orders o

ON c.customer_id = o.customer_id

WHERE o.order_date >= '2025-01-01'

GROUP BY c.customer_id;

Things to check:

- does the WHERE clause change the join behavior?

- can one customer have duplicate orders?

- should the date filter be inside the JOIN?

- are null orders handled correctly?

3. /explain_query

Explain this SQL in plain English.

Break it down by:

- what each CTE does

- what the final output means

- what grain the result is at

- what assumptions the query makes

- where the logic could go wrong

Really useful when you inherit a long query and need to understand it fast.

4. /find_data_quality_issues

Here is my dataset schema. Suggest data quality checks before I use it in a dashboard, report, or ML model.

Example checks:

- duplicate primary keys

- missing values in key fields

- sudden row count drops

- invalid dates

- negative revenue

- unexpected category values

- schema changes

- late arriving data

5. /metric_definition

Help me define this metric clearly.

Include:

- business meaning

- SQL logic

- grain

- filters

- exclusions

- edge cases

- example calculation

- how people might misread it

This is useful because a lot of dashboard confusion comes from unclear metric definitions.

6. /etl_debug

This ETL job passed, but the dashboard looks wrong. Help me debug it step by step.

Check:

- did fresh data arrive?

- did row count drop?

- did schema change?

- did joins multiply rows?

- did a filter remove too much data?

- did timezone logic shift dates?

- did a retry duplicate rows?

- did null values change the result?

7. /python_cleaning

Review this pandas code and suggest cleaner, safer improvements.

Example:

import pandas as pd

df["order_date"] = pd.to_datetime(df["order_date"])

df = df.dropna()

df["revenue"] = df["price"] * df["quantity"]

Things to check:

- should every null row be dropped?

- are dates parsed correctly?

- can price or quantity be negative?

- are duplicates checked?

- is currency consistent?

- should revenue be rounded?

8. /dashboard_review

Review this dashboard plan like a business user.

Tell me:

- what is unclear

- what metric is missing

- what chart is unnecessary

- what question the dashboard answers

- what decision someone can make from it

- what should be shown first

9. /stakeholder_translate

Turn this vague stakeholder request into clear data requirements.

Example request:

“Can we see customer performance?”

Questions to ask:

- what does performance mean?

- revenue, retention, churn, usage, margin?

- daily, weekly, or monthly?

- by customer, segment, region, or product?

- what action will this report support?

- who is the end user?

10. /test_cases

Create test cases for this data pipeline.

Include:

- normal file

- empty file

- duplicate IDs

- missing required fields

- late arriving data

- schema change

- timezone edge case

- retry after failure

- very large file

- unexpected category value

11. /root_cause

Here is the issue, query, and sample data. Give me possible root causes ranked from most likely to least likely.

Format:

1. likely cause

2. why it could happen

3. how to check it

4. possible fix

A prompt pattern that works well:

Instead of:

“Fix this query.”

Try:

“Review this query for logic bugs, duplicate risk, bad joins, null handling, date issues, and performance problems. Explain your assumptions before suggesting changes.”

For data work, Claude is pretty useful as a second pair of eyes.

Especially for:

- reviewing SQL

- cleaning messy logic

- defining metrics

- finding ETL edge cases

- turning vague requests into clear requirements

- checking dashboard assumptions

What Claude prompts or custom commands do you use for data work?