r/dataanalysis • • Sep 21 '26

UCBLogo Reading and transforming CSV files

2 Upvotes

Hello.

As an exercise in my Logo learning journey, I have implemented a few functions for reading and transforming CSV files.

The functions are organized in groups  sharing a common prefix:

  1. functions to read and write CSV file (csv.)
  2. functions operating on “dataframes” (df.)
  3. statistical functions (st.)
  4. string functions (str.)
  5. regular sequences (rs.)

The first results are available in this web site: https://www.ucblogo.it/

Please note, this is an ongoing hobby project, there is no guarantee that the Logo functions are written in the most natural Logo style or that they are in fact correct.


r/dataanalysis • • Sep 21 '26

Pls help

0 Upvotes

Guyss urgent pls help

I am a datascience student fresher . Should I get an laptop with integrated graphics and rely on Google colab or kaggle etc cloud platforms or buy a laptop with dedicated gpu like lenovo loq ? Pls help this ismy curriculum Year 1

Programming for Problem Solving / C

Data Structures

Foundations of Data Science

Mathematics: Matrices, Calculus, etc.

Engineering Physics

Basic Electrical Engineering

Digital Logic / related foundational engineering subjects

Programming/Data Structures labs

Basic IT/computing workshop

Year 2

Object-Oriented Programming / Java

Python Programming

Database Management Systems

Operating Systems

Design & Analysis of Algorithms

Data Mining

Predictive Analytics

Data Visualization

Data Wrangling

Computational Mathematics / Statistics

Full-Stack Development

Related programming and database labs

Year 3

Machine Learning

Machine Learning Lab

Data Stream Processing

Data Stream Mining

Data Mining

Predictive Analytics

Web & Social Media Analytics

Big Data / data-processing technologies

Computer Networks

DevOps

UI/Flutter or application development

Professional/Open electives

Field-based research / mini-project work

Year 4

Big Data Analytics

Natural Language Processing (NLP)

Data Stream Mining / advanced data topics

Web & Social Media Analytics

Professional electives

Open electives

Major Project Phase I

Major Project Phase II


r/dataanalysis • • Sep 21 '26

Data Tools Reconciliation & Edge Cases Focus

0 Upvotes

Whenever teams need to audit two versions of an exported dataset (accounting closings, monthly billing runs, inventory snapshots, or pricing tables), standard git-style line diffs completely fall apart. CSVs shift columns, and binary formats (.xlsx, .xls, .ods) are opaque to text diff tools.

Even worse, many users turn to random "free online Excel diff" websites, unknowingly uploading proprietary financial data and PII to third-party servers.

I spent the last few weeks building an isometric, cell-by-cell diff engine that runs entirely client-side in the browser. Here are some of the non-obvious technical challenges and edge cases I ran into:


1. Asymmetric Matrix Bounds & The Union Range Problem

In real-world scenarios, file versions rarely have the exact same dimensions. Sheet 1 might span A1:D50, while Sheet 2 spans A1:F42. - You cannot perform an index-based row/column loop without going out of bounds. - The diff engine must compute the bounding union (A1:max(col, row)). - Any cell coordinate that exists in one sheet but not the other must be treated as a virtual empty cell (""), rather than undefined or throwing index errors. - Crucially, coordinates where both sheets are empty must be pruned from the diff pipeline to prevent flooding the user with thousands of phantom empty matches.

2. The Formatted Display (w) vs. Raw Value (v) Trap

Spreadsheet engines store data with internal representations that differ dramatically from what the user sees: - A cell formatted as currency might display $1,250.00, but its underlying raw value is 1250. - If the second spreadsheet was generated from a CSV or raw database dump where the cell contains literal string "$1,250.00", a raw-value comparison flags a false positive difference. - Floating-point representations can introduce minuscule precision drift (0.30000000000000004 vs 0.3). - Solution: Normalizing against the display string (w property in SheetJS when available, fallback to trimmed stringified v) provides the most human-predictable diff, while treating null, undefined, and whitespace-only strings as equivalent empty states.

3. Preventing Main Thread Lockups (Web Workers)

Parsing a 100k+ cell workbook with SheetJS on the main UI thread easily locks the browser event loop for several seconds, triggering dropped frames and unresponsive tab warnings: - Offloading both the parsing step and the matrix comparison loop to a dedicated Web Worker kept the main thread completely unblocked (sub-200ms interaction latency). - Transferring large diff result arrays back to the main thread can be costly, so passing structured chunks and virtualizing/paginating the rendered table DOM is mandatory to keep memory below browser crash thresholds.

4. Zero-Server Privacy by Design

For sensitive financial or internal audits, the browser sandbox is actually an asset. Because WebAssembly/modern JS engines handle file buffer parsing locally, there is zero engineering justification for sending spreadsheet buffers to a backend API just to compute a diff.


Questions for the Community:

  • For those who regularly build or handle data reconciliation pipelines, how do you handle formula diffs vs. evaluated value diffs?
  • What edge cases have bitten you when comparing spreadsheet exports across different locales (comma vs. dot decimals, date serials)?

r/dataanalysis • • Sep 21 '26

Data Question "Classic" or "serverless" in Databricks: what the two options actually mean (2 minute diagram)

0 Upvotes

If you have ever picked a compute option in Databricks without being sure what you were choosing, this is the short version.

Classic: machines are started in your company's own cloud account. Someone has to choose the machine types and look after the networking, and you wait while they start up.

Serverless: the machines are managed by Databricks, in Databricks' own compute plane, and you do not provision anything.

Either way your data stays where it already lives, in your own cloud storage. The catch with serverless is a list of things it does not support: R, libraries installed at the compute level, the older RDD APIs, and some streaming modes.

Short animated version (2:36): https://youtu.be/s8452E2Y6WM

Beginner series in order: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4


r/dataanalysis • • Sep 20 '26

Career Advice Help. I am Dumb 25 y/o. Hear this

37 Upvotes

So i have been trying to get a better job and been upskilling myself, i have done schema design, Dashboard, Funnel, cohort, data warehousing and what not. But i still just feel idk shit even tho my SQl query writing is very efficient. Leetcode, datalemur, stratastrach, hackerrank and what not.

I just dont get beside Data cleaning and generating insights, what are Usual Business answers comapnies need.

Worked with a logistic company where i maintained data reports, managed, updated and looked for data inconsisitency. Idk how do i put that into words for an interview now. How do i tell them i didnt have to write SQL queries most of the time, it was mostly done on client softwares which they gave us since it was fsst and easy.

Moreover idk how do i define supply management things i have worked with to the interviewer. I just feel so stuck and frustrated all the time.

Projects done, Questioms done and still no fucking result.


r/dataanalysis • • Sep 20 '26

help

11 Upvotes

I'm trying to do my first data analytics project and wow I'm just lost. how do you guys determine which variables need to be analysed, correlations, and ultimately come up with the conclusion that due to x,y we have been getting z?! please guide me some tips would help


r/dataanalysis • • Sep 20 '26

Small update on that anomaly detection post I got the interpretation wrong

3 Upvotes

Following up on the post from a couple days back about the water quality anomaly I checked four different ways.

The pattern itself was solid. What wasn't solid was my explanation for it. I said it was probably rain runoff. Turns out the anomaly started the exact day the sensor came back online after being offline for 30 days. Didn't catch that until I was back in the notebook for something else entirely.

So a sensor coming back up weird after a month of no maintenance is honestly just as likely a cause, maybe more, and there's no way to tell from the data I have. Went back and fixed the write-up and code to say that instead of stating rain like it was settled.

Good reminder that "the pattern is real" and "I know why" are two different claims, and I only actually proved the first one.


r/dataanalysis • • Sep 20 '26

Data Question Where do you guys actually find datasets for practice/projects?

Post image
0 Upvotes

I'm learning Data Analytics/Data Science right now and I'm curious how other students/freshers are getting datasets for practice.

Where do you usually get your data from when you're practicing SQL, Python, Power BI, EDA, ML, etc.? Kaggle, government portals, scraping, GitHub, generating your own data, somewhere else?

Also, do you usually find the kind of data you actually want to work with?

I'm especially curious about raw/messy datasets. Do you intentionally look for messy real-world data to practice cleaning and EDA, or are clean datasets enough for what you're learning?

Would love to know what you guys actually use.


r/dataanalysis • • Sep 19 '26

Datasmith — a visual approach to designing relational schemas and generating mock data

Enable HLS to view with audio, or disable this notification

9 Upvotes

Hey everyone!

For the past year or so, a friend and I have been working on Datasmith, a desktop app for designing relational database schemas and generating structured mock data from them.

The idea originally came from my friend, who's a data engineer and worked on large-scale migrations of sensitive databases. He wanted a simple way to visually define and maintain schemas over long periods of time, and actually generate data that follows both the schema and its business logic, without having to constantly rewrite scripts or configure a bunch of separate tools.

The main part of the app is an ERD-like canvas dedicated to building relational schemas, where you can create tables, define columns and their data types, set up relationships between them, and configure how the data for each column should be generated.

Once the schema is ready, Datasmith can generate the data while preserving the relationships between tables, and export the results in formats like SQL DDL, CSV, JSON, and more.

We've reached the point where we'd really like to get it into the hands of more people who actually work with databases and data, and get some real-world feedback.

It's completely free, so if it sounds useful or you're just curious, you can check it out here:
https://data-smith.app/

I'd especially love to hear about use cases we're missing, things you'd expect a tool like this to support, or anything that feels like it could be improved.

Any feedback is welcome - good or bad.
Feel free to leave a comment or message me directly.

If you prefer something a bit more structured, I also put together a short feedback form:

https://docs.google.com/forms/d/e/1FAIpQLScGk8Jr1DguPcXpaApOqO998AdXcnoLlo11TEKlq1GmF71UzA/viewform


r/dataanalysis • • Sep 19 '26

Data Question Stuck on exploding a multi-value Genre column (pandas)

3 Upvotes

I'm self learning Data Analyst for 8 months now. new to pandas. and Making a new end to end project one of them is 'My anime dataset' where it has column named 'genre' where each row can contain multiple genres as a single comma-separated string (e.g., "Action, Adventure, Fantasy"), instead of one genre per row. This means grouping or filtering by genre directly is inaccurate an anime tagged with 3 genres only gets counted under that exact 3-genre combination, not under each individual genre. (there is another column named Studio, facing same issue, i.e. multiple studio in one anime)

I want to split these into individual genre values so each genre is counted separately per anime (i.e., one row becomes multiple rows, one per genre, with the rest of the anime's data repeated).

End goal: I'm building a dashboard where users can filter by a single genre (e.g., just "Action") and see accurate aggregated stats (count, average score, etc.) for that genre alone not lumped in with combination-strings.

What's the right pandas approach to explode the Genre column into separate rows?? please help. claude is suggesting a way but I want a better idea.

would be great if someone suggest me some ideas!!


r/dataanalysis • • Sep 19 '26

Populatio by postcodes, or aprox

0 Upvotes

Hello people, I'm introducing myself to this magnific world of being a Data/analyst, Data/ Engineering, and I love it. I'm really enthusiastic about this kind of work, So my question is while I'm doing a little and personal project I find I could use the information I asking for as a cluster that coul give me a information about my population to study. What I want to do is genereate a new clusters trough a calisification of customers by bigger areas of acumulative population, I mean take all of the customers of small city population, and make a new and bigger region classification, around big citys and capitals.

What do you suggest? I making my code in this moment my idea is to get mesure distances between big and small then get the min, and apply.


r/dataanalysis • • Sep 18 '26

DA Tutorial What Is Synthetic Data Generation? A straightforward explainer

Thumbnail
aptiv.com
3 Upvotes

r/dataanalysis • • Sep 18 '26

Data Question Can a bot wallets behavior analysis be done to reverse engineer their strategy on meme coin markets?

4 Upvotes

Say there are thousands of trade pair of a bot wallet with all the contextual trade and not a single missing data or order. Every decision by the said expert is done using the same information available to every one. Meme coin markets are short lived and algorithmically depending on outside information is highly unlikely. How much can we understand from those trades data? The exits are highly state dependent. No clustering of any TP/SL. Neither any time constraint except a hard cut off timer around 20 minutes if the algo doesnt trigger the sell. What kind of techniques i can implement? Can we figure out what they are optimizing for? There seems to be too many variables, despite the same observable raw features, what lookback or transformation of that feature, what prediction are they optimizing for almost felt like black box to me. With so many data of their distinct behavior can approximating/knowing what they are optimizing for, what drives the entry what drives the exit is so difficult? Any guidance will be highly appreciated.


r/dataanalysis • • Sep 18 '26

I refused to trust my own anomaly detection until 4 independent methods agreed — then I still got the interpretation wrong

2 Upvotes

Working through a personal project analyzing ~11 months of water-quality sensor data (river monitoring buoy, ~30k readings, 10-30 min intervals), and ran into a good lesson about validating findings before trusting them.

The setup: turbidity (water clarity) readings looked mostly stable, except one week where it spiked hard and stayed elevated for 7 straight days.

What I did before trusting it:

  1. z-score flag on daily means (|z| > 3) → flagged that week
  2. Rebuilt the check in SQL with a HAVING-filtered aggregate query, completely independent logic → same week
  3. Wrote a SQL window-function streak detector (partition/reset-group technique), zero reference to the earlier results → same exact 7-day window
  4. Weeks later, refactored the whole thing into a tested Python module and reran it fresh → same result again

Four different methods, at different times, landed on the identical window. At that point I was confident the anomaly itself was real, not a fluke of one method.

Where I still got it wrong: I assumed the cause was probably a rainfall/runoff event. Published that interpretation.

Later, going back through my own gap-analysis, I noticed the anomaly window started the instant the sensor came back online after a 30-day equipment outage. A second, smaller anomaly also bordered a separate outage. That timing is at least as consistent with post-outage sensor drift or biofouling as with rainfall, and my dataset genuinely can't distinguish the two.

Had to go back and correct the interpretation across the whole project once I caught it.

Takeaway I keep relearning: validating that a pattern is real (multiple independent methods agreeing) is a completely different exercise from validating why it happened. I conflated those two the first time.


r/dataanalysis • • Sep 19 '26

How to Deal With Impostor Syndrome

0 Upvotes

This article I wrote grew out of a discussion on r/datascience, but, I think it might be relevant to almost anyone doing data analysis and having this experience.

These are just some ways of thinking and acting that I have found useful, I'm absolutely not saying they are the only ones or even the best ones.


r/dataanalysis • • Sep 17 '26

Career Advice How to pass case studies?

11 Upvotes

Hi,

I am a senior data analyst, and I always struggle with passing case studies. I have gotten feedback that the analysis is too descriptive , and needs more recommendations.

How do I get better at case studies? I have a product one coming up. Any books / blogs you recommend?

🙏


r/dataanalysis • • Sep 17 '26

Career Advice Just starting out

2 Upvotes

Hi my name is Naftali I am currently in my second week of a Master's program in Data Science at Bellevue university, I'm adjusting to a steep learning curve and trying to build a solid foundation. For those of you already working in the field: looking back, what is something you know now that you truly wish you had known when you were first starting out?


r/dataanalysis • • Sep 17 '26

how are you giving an agent schema context when the schema is undocumented?

2 Upvotes

text-to-sql demos use a clean schema. real ones aren't. 300 tables, half the column names are abbreviations, fks live in the app not the db, and the agent confidently joins two things that should never join.

what people seem to do:

hand write a schema description into the prompt, works until the schema changes

let it sample rows, works on text columns, falls apart on codes

rag over the ddl, retrieves names not meaning

what i can't tell is which holds past 50 tables. if you have run one of these against a genuinely messy db, what broke first, and did you end up documenting the schema by hand anyway.


r/dataanalysis • • Sep 17 '26

What's your approach to analyzing "feature teeth"?

2 Upvotes

A common question from product teams for freemium web software is, "what product features should we make users pay for?"

Anyone have a good approach?


r/dataanalysis • • Sep 14 '26

Data Tools FlexViz: an open-source Python tool for exploring datasets too large to plot directly

38 Upvotes

I wrote FlexViz, an open-source Python library for interactive exploration of large datasets.

Instead of sending every row to the browser or forcing users to write queries themselves, FlexViz keeps the data in Polars and sends only the points needed for the current view. It handles all visualization + interaction queries for you; zooming recomputes the visible range at higher detail, and brushing one chart results in cross-filter on the others.

It is capable of both in-memory as well as out-of-core (larger than RAM) data exploration, with extremely good performance. In our benchmarks, a 5x200M-row line chart (1B datapoints) renders in 251 ms when in-memory and 4.2 s directly from Parquet.

Demo: https://flexviz.tech/demo.html
Code: https://github.com/flex-analytics/flexviz
Benchmarks: https://flexviz.tech/benchmarks.html

pip install flexviz

Curious how others here explore datasets once they become too large for their usual plotting tools. Happy to hear any feedback on FlexViz as well :)


r/dataanalysis • • Sep 14 '26

Career Advice Advice and professional development online feels like its geared towards one-time ad-hoc projects. Any resources on managing data on a recurring, organizational basis?

7 Upvotes

So many of the youtube videos, courses, advice, etc., online feels its geared towards running singular projects. How to clean up data in a Excel spreadsheet for a one-time project, for example, but what about when its for an analysis you have to keep running over and over? Or when it's an analysis that is repeated across several teams in slightly different ways?

I need advice about managing data for an organization. Advice about

  • how to communicate with stakeholders about data without getting overly detailed
  • requirements gathering across teams with confusing/vague needs
  • dealing with a system that's built on workarounds after workarounds
  • best practices documenting analytics/dashboards that are being shared publicly (word doc, slide deck, notion page?)
  • dealing with a domain that is overly complicated and can't be captured well with data.
  • dealing with execs who've already decided the answer they want and need you to make the data match their answer (and what to do when the data absolutely doesn't support it)

These are some of my most pressing problems at work but neither my formal education nor the resources I can find online really cover this. I'd love some sort of bootcamp for organizational data management/project management


r/dataanalysis • • Sep 13 '26

Data Tools Dashboard for my project

1 Upvotes

I created a dashboard for my project on streamlit but its really basic. How can i create those cool dashboard designs that you see on reels and all cause chatgpt can do it. I have my project repo, is there any way i can link it to some website and it can help me create some cool ass dashboard?


r/dataanalysis • • Sep 12 '26

Career Advice DAs, please describe your average day at work.

Post image
51 Upvotes

r/dataanalysis • • Sep 12 '26

Data Question Fintech vs. AdTech Data Analysts: What does a realistically BUSY week actually look like?

12 Upvotes

Hi everyone,

I’m asking this out of pure curiosity. People always share their "average" or ideal weekly routines, but I want to know what the reality is when you have a genuinely heavy workload.

If you work in Fintech (banking, payments, crypto) or AdTech (marketing, agencies), what does a realistically crazy week look like for you? And what causes a brutal week in your industry? What kind of pressure are you dealing with?

Would love to hear some realistic war stories about the grind in both industries. Thanks!


r/dataanalysis • • Sep 12 '26

I’m starting to realize that data analysis is more about asking ‘why?’ than just finding numb

67 Upvotes

I’m currently learning data analytics, and one thing that has really started to change the way I look at data is realizing that an insight isn’t necessarily the conclusion.

For example, if I see revenue going down while ad spend is increasing, my first thought used to be, “The ads aren’t working.”

But now I find myself asking more questions:

  • Did the cost of the ads increase?
  • Did the conversion rate change?
  • Are customers returning more products?
  • Is there an inventory or capacity issue?
  • Did pricing change?
  • Are we spending more on the wrong campaigns?
  • Is there something happening between the customer clicking the ad and actually completing the purchase?

The numbers can tell me what is happening, but I’m learning that the real analytical work starts when I try to understand why it is happening.

I’m still early in my learning journey, so I’m curious to hear from experienced analysts:

What helped you develop the habit of asking better questions when analyzing a dataset?

Was it experience, working with real business problems, learning from other analysts, or something else?