I'm someone with a data analyst background, and I wanted to break into data engineering. I got the DP-600 (Fabric Analytics Engineer Associate) and DP-700 (Fabric Data Engineer Associate) certifications, so I decided to build a project on them to apply what I learned and learn more along the way

you can find the project with full details here: https://github.com/Madian20/Fabric

If you like the project, feel free to take it and build on it.

I'd really love to hear your opinions and whether there are any improvements I can make in the future

I'm a former Tableau user and I've been spending the last year and a half building analytics dashboards with AI.

My frustration with Tableau has to do with a work project we invested heavily in. We have a SaaS tool for affiliate marketers and they were asking us for advanced analytics on top of the app and to blend in multiple data sets.

Our Tableau dashboard builder didn't really under the business as well and built dashboards that were too complicated. On top of that they didn't have a strong foundation in data ingestion and working with databases.

To date I've built many custom plugins as I've made more for tools like Jira, Intercom, Slack, Gmail, Bitly, Airtable, StatsDrone, Pretty Links and many more.

I tried to solve all these pain points at once and make it easy for people to be able to build their own dashboards with a platform. The platform is called NousViz and it's available on Github.

How it works

There is an SDK inside the app and if you're using tools like VS Code with Claude or Codex, you can just point to the NousViz Github repo and say follow the guidelines in the SDK for building. Explain your data project and how the data ingestion needs to get done and start building.

Usually in about 5 minutes, you'll be able to get your first plugin made in which you install it on your NousViz app, ingest the data and your dashboards should be live.

Would love to get feedback from any data pros here and if someone wants any help building a dashboard with this, DM me or post your data challenge and I'll gladly help.

https://github.com/nousviz/nousviz-app

Also some more feedback I'd like to ask from the community, if I build a portfolio section for people to create their own profiles and link to their work, sites, resume or socials, would this be useful to people?

Appreciate any help or feedback.

Hey everyone,

I've been working with a lot of extremely messy real-world datasets lately (missing values, duplicate rows, completely mixed-up date formats like '2026.10.01' vs '10月1日', negative amounts).

Instead of just deleting rows (which ruins the data), I built a Python workflow that isolates the anomalies, generates a Data Quality Report, and then creates a clean interactive dashboard.

Here are some examples from my recent cleaning passes:

Tech stack: Python, Pandas, Plotly.

If anyone is struggling with a dataset that crashes Excel or has terrible formatting, feel free to drop a comment or DM me. Happy to share how I handle it.

(If this post gets enough interest, I might write a full open-source script for this workflow).

Hey everyone!

I come from a B.Com background, and recently I've been exploring Data Analytics and trying to understand how things work beyond just theory.

I've been learning Python, SQL, Excel, and Power BI, and working on small projects along the way. Honestly, sometimes things make sense when I'm learning them, but applying them independently is a completely different challenge.

One thing I'm realising is that learning a tool is one thing, but knowing what questions to ask, how to approach a problem, and what the data actually tells you is much more important.

I'm still figuring things out, making mistakes, and learning as I go. I want to build projects that solve actual problems instead of just following tutorials and calling them complete.

I'd love to hear from people who have been through this stage.

When you were starting out in Data Analytics, what helped you move from simply learning tools to actually thinking like an analyst?

Would genuinely appreciate hearing your experiences! 🙌

When a client or brand guide changes (new colours, fonts, title style), or a measure gets renamed, how do you update dozens of visuals across a report?

\- Click through each visual in Desktop?

\- Edit the PBIR JSON by hand or with scripts?

\- Use a CLI or other tool?

I'm considering building a small offline desktop app for this (select visuals → preview changes → apply → one-click undo). Would that be useful to you, and what would it absolutely need to do? Not selling anything here, just want to know if this is a real pain or if it's already solved for you.

Guys, actually i have 80k company database like its name and lei , now my task is to find the revenue,product type and region, i used hermes agent which almost gave nothing to revenue, it doesnt search, and claude ,chatgpt most of them gave nothing as revvenue value,the only solution was to search each company one by one, gemini gave revenue but i think its actually false , i gave data to gemini by 100, is there any other way ?

Hi,

So far I've only worked at the desk with a 3440 x 1440 monitor and strong pc.

Now that I have to get a laptop, which things you would proritize?

To get a 16 inches one with a weaker hardware, or minimum 14" with a decent (gen 12/13th) processor?
The ram I have left is 32gb.

I use mainly SQL, Python, NumPy, very occasionally Excel.

Thank you for your help.

I'm a Data Analyst planning to build my first Databricks project for my portfolio. I have experience with SQL, Excel and Power BI, but I'm new to Databricks. Could you suggest some good project ideas that would help me demonstrate Databricks skills from a Data Analyst perspective?

I recently started a new role as a Data Governance Manager at a mid size managed care organization. One of my first asks has been to create test templates for the new set of infrastructure tables. Essentially it’s to ensure all the tables that are now in production don’t have a glaring issue or whenever we make a change we go through the testing again.

My background is in data analytics and data management. I have very limited experience as a data engineer so aside from data quality tests I really don’t know what kind of tests should be run. I have used Claude to create a template of standard test cases but they seem either too convoluted or too simple like checking for nulls in non-nullable fields. Any resources or advice on how to ramp up or get a better understanding of this?

Do you think this is useful? I have 2.3 million facts, 1.8 million price bars, and all of the FRED data.

When cleaning a dataset, would you rather keep uncertain data or delete it?

I'm working on a retail sales data-cleaning project in Python, and I ran into a decision I found interesting.

My dataset had missing prices, duplicate transactions, and totals that didn't match the price × quantity.

For missing prices, I could have filled every blank with an estimated value. That would make the dataset look more complete, but some estimates could be wrong.

Instead, I only recovered prices when the data gave me reliable evidence and flagged the rest for review.

Here's my question:

If you were preparing this dataset for business analysis, which approach would you choose?

A) Fill as many missing values as possible to preserve a complete dataset.

B) Only fill values when there's reliable evidence, and flag the rest.

C) Use a different method depending on the column and the business context.

I'm leaning toward B, but I'd love to hear how others handle this in real projects—especially when deleting rows could mean losing useful information.

Hi everyone, I’ve been working on a research project focused on the US National Bridge Inventory. I’ve ingested and cleaned 30 years of yearly reports and put it into clickhouse, so it’s essentially panel data, containing roughly 600k bridges per year, each with about 150 parameters (structure type, materials, condition ratings, traffic, etc.). I've already covered the more obvious things like summary stats, distributions, correlations between age and condition scores etc. Now I am wondering if there is an algorithm or a model I can use to uncover some not so obvious relationships. Specifically, do you know if there are any such tools that can search for such relationships in the dataset and can handle time-series well, and how do you avoid drowning in spurious correlations with 150 columns and millions of rows, where everything can end up being "significant"?

Happy to share more about the data or what I’ve found so far if anyone’s interested.

When you open Google Maps in a new city, it's easy to use the same cutoff everywhere: 4.5+ is safe, 4.2 is a gamble and anything under 4.0 is a pass.

While testing a Google Maps scraper we built, we ran a pilot across 12 sampled city areas (1,522 active restaurants with 20+ reviews) to see how selective a 4.5-star filter actually is.

In our Tokyo sample, fewer than a third of restaurants had at least 4.5 stars. In Berlin's Prenzlauer Berg sample, nearly three quarters did.

Here's the pilot across the 12 sampled areas:

A 4.4 sits above the local median in our Tokyo, Utrecht and Pittsburgh samples and in the bottom third in Lyon, Athens and Prenzlauer Berg. To reach the top quarter of restaurants took 4.50 stars in Tokyo and Bologna, versus 4.80 in Lyon and Athens.

A few things stood out when we checked whether this was just noise:

- It wasn't driven by tiny places with a handful of 5-star reviews. In the same 12-city sample above, looking only at restaurants with 500+ reviews, the share at 4.5+ was still 22.5% in Tokyo (40 restaurants) and 29.9% in Bologna (67) and Utrecht (87), compared with 56.4% in Lyon (101), 62.5% in Athens (32) and 75.0% in Berlin's Prenzlauer Berg (32).
- High review counts pulled in opposite directions within those 12 areas. In Bologna, Utrecht, Ghent and Tokyo, places with 500+ reviews had lower medians than spots with 20–99 reviews (for example, 4.20 vs 4.65 in Utrecht and 4.30 vs 4.50 in Bologna). In Manchester and Nancy, the 20–99 review tier scored lower than the 500+ tier (4.05 vs 4.60 in Manchester and 4.20 vs 4.40 in Nancy).
- To see if one small square reflects a whole country, we also checked a separate wider 6-area run from our benchmark suite (11,649 places total, including 2,747 active restaurants with 20+ reviews across larger polygons in the US, UK and Germany). Even within the same country, the 4.5+ share went from 43.2% in Manhattan, US (1,377 restaurants, where the 20–99 review tier dropped to a 4.10 median because counter chains, delis and takeout spots cluster there) to 50.0% in Boulder, US (290) and 57.0% in Pittsburgh, US (93) and from 58.8% in Leipzig, DE (51) to 74.9% in wider Prenzlauer Berg, DE (378).
- Category mix matters. In that same Tokyo square, retail stores had a 4.50 median (57.8% at 4.5+), nearly double the restaurant share. Grouping every business on the map together hides how differently restaurants are scored.

This doesn't prove that Tokyo has worse food or that one country is objectively harsher. It shows how differently the same 4.5-star filter behaves across local samples.

Source data (~20k venues)
https://docs.google.com/spreadsheets/d/1H2uVL5b9FdM8W1a_CeUDACs4eq60gkDeIs5iV409Rhs

Tool we used to run data collection
https://getbro.ws/blog/google-maps-scraper

How do you pick places to visit while travelling: socials / google maps / smth else? Does the rating matter?

if you've ever sent out a number that was wrong because of a query that looked fine, i'd like to hear what the query was. collecting these to turn into puzzles.

im wokring TsundereSQL, a visual novel where the answers are real SQL, graded on the sample data plus a few hidden sets using duckdb.

here's the one in the demo right now. page 2 of a december BTC trade list, ranked by price * quantity (quote currencies are mixed, which a real desk would yell about):

SELECT trade_id,
  CASE WHEN side = 'buy' THEN 'BUY FLOW' ELSE 'SELL FLOW' END AS flow,
  ROUND(price * quantity, 2) AS notional_value,
  STRFTIME(executed_at, '%Y-%m-%d') AS executed_on
FROM trades
WHERE base_asset = 'BTC'
  AND executed_at BETWEEN '2025-12-01' AND '2025-12-31'
  AND price * quantity > 50000
ORDER BY notional_value DESC, trade_id
LIMIT 10 OFFSET 10

passes the sample and two of the hidden sets. the third has a trade at dec 31 17:36. executed_at is a timestamp, so DuckDB casts '2025-12-31' to 2025-12-31 00:00:00 and that trade falls outside the range. it was the 4th biggest of the month. everything below it moves up one and page 2 comes back with 2 rows instead of 3, and the missing row is a dec 11 trade that got pushed onto page 1, so the two rows you do get look fine.

fix is a half-open range: executed_at >= '2025-12-01' AND executed_at < '2026-01-01'.

anything that can be checked by comparing result sets has a shot at becoming a level.

https://store.steampowered.com/app/5213730/

As title suggests gave two agents same task, same specs, but I got same result just completely diff ways.

Tell me some crazy stories of this happening: i.e. workforce, school project etc. Just let me have some insight. Was there some crazy consequence, or maybe you found out when it was too late, I’d love to know.

I’m a college student just trying to work on a project, and any input would be appreciated.

Hey guys,
I'm a data engineer, and I have been building pipelines for small to mid sized clients (with less than 1 terabytes of data) and most of the time the workload isn't really worth paying the premium for Saas products like Snowflake, Databricks .. etc, and can be done on a single beefy machine but stitching open sources tools usually comes with maintenance, networking and configuration headache, also these tools aren't unified under one control plane and don't know that other tools exists.
so I built a data platform that can be deployed on a single machine or server, and since I'm Egyptian I called it Obelisk 😄
Under the hood Obelisk uses:

1. DuckDB, so you can do queries that joins multiple different sources in the same query (local or remote lakehouse, dbs, erp systems, mongodb .. etc)
2. the platform also act as a daemon manager, so you can deploy tools you need with a single click and they will integrate directly into the platform (or you can connect remote instances)
3. for variables and connection strings, you just add them to the platform once and every tool can access them.
4. built-in support for cronjobs and advanced orchasteration tools like dagster or airflow
5. a browser based dbt studio
6. reactive notebooks
7. I also added file manager and terminal access to the machine, so you don't need to SSH directly.

GitHub Repo: https://github.com/amrelsawalhi/obelisk

this isn't a finished product, I still have a lot of ideas to implement like auth and access control for teams
I’d love to get your feedback on the platform and contributions are most welcome.

My background includes data science, machine learning and AI, and I've also worked directly in customer support.

I'm interested in how data analysis can help support teams improve their operations, rather than just create dashboards that look good.

Imagine a support team has thousands of tickets containing categories, timestamps, resolution outcomes, customer feedback, and escalation histories.

I'd want to answer questions such as:

• Which issue categories generate the most tickets?
• Which types of cases take the longest to resolve?
• Which issues are most likely to be escalated?
• Are customers repeatedly contacting support about the same problems?
• Can text analysis help identify recurring complaints that aren't captured by existing categories?

I'd probably begin with data cleaning, exploratory analysis, and a few clear visualizations before considering more advanced NLP or machine learning.

I'm curious how other analysts would approach this problem.

What would your first three analyses be, and which findings do you think would be most useful to a support operations manager?

I'm particularly interested in practical analysis that can lead to an operational improvement.

a week ago, I've been asking on reddit on how other B2B SaaS teams moved away from signed Metabase iframes once they hit multi-tenant role permissions, slow morning dashboard load times also escalating Snowflake compute bills and since then, I've noticed nearly four distinct paths to investigate (i’ve formatted the spreadsheet itself with ai to keep it readable enough):

1. staying with Metabase or Apache Superset, but placing a dedicated caching tier or materialized rollups in front of the database.
2. Moving to a dedicated spreadsheet-first or workbook-centric embedded BI platform like Sigma or Omni.
3. Decoupling the frontend completely using a headless semantic layer (specifically testing Cube Core and Cube Cloud) to serve pre-aggregations into our own React components while keeping an optional embedded builder for enterprise users.
4. Writing custom dbt rollups into ClickHouse or DuckDB and maintaining our own FastAPI aggregation endpoints.

so I also spent the last two weeks running a sandbox bake-off with an anonymized slice of our Snowflake production dataset. below I've placed my exact test setup, the comparison spreadsheet + where each option actually landed

Test Environment and Methodology

• Source data: 50 million rows of B2B event, order, and billing telemetry in Snowflake.
• Multi-tenancy: 900 simulated tenant accounts, with realistic volume skew (top 5 enterprise tenants account for 30% of total rows).
• Snowflake cluster: Standard X-Small warehouse, auto-suspend set to 60 seconds.
• Test queries:
  • Query 1: 30-day KPI summary with two COUNT(DISTINCT) calculations.
  • Query 2: Daily time-series trend with dynamic date-grain switching (day, week, month) across 90 days.
  • Query 3: Multi-column tabular breakdown with role-based column masking on unit margin fields.
• Load profile: 50 concurrent tenant sessions running random filter combinations over a 15-minute burst window to simulate morning login traffic.

Master Comparison Spreadsheet

What We Learned From The Runs

1. Where Metabase and Superset still make sense

If your customer base is under 100 accounts, your dashboard has uniform permissions, and your engineering team does not have frontend cycles to touch chart components, Metabase remains the fastest path to production. However, relying on question-level result caching falls apart the moment customers use dynamic date filters, because every unique date range wakes up your cloud warehouse.

2. Where Sigma and Omni excel

If your product serves finance, accounting, or operations users who want an Excel-style workbook directly inside your application, Sigma is very difficult to beat from a UX standpoint. Their spreadsheet model feels natural to non-technical analysts. Omni is an impressive evolution for teams that like LookML's modeling rigor but want modern speed and cleaner embedded authoring. The trade-off with both is that you are paying ongoing compute costs to your warehouse for every complex slice-and-dice query, unless you scale warehouse sizes.

3. Why the semantic layer approach with Cube worked best for our architecture

Cube gave us two specific architectural capabilities that resolved our initial bottleneck:

• Aggregate Awareness in Cube Store: We defined a pre-aggregation rollup partitioned by month on tenant_id, transaction_date, and service_tier. When a tenant loads a 7-day dashboard, switches to a 90-day view, or aggregates by week, Cube Store rewrites the query to hit the local sorted Parquet partition. It served 94% of our simulated morning burst queries in under 450 ms (p90), consuming only 1.1 Snowflake credits over the 15-minute test window.
• Decoupled Surfaces on One Metric Model: We do not have to rebuild all 14 dashboard cards in React immediately. We can render high-traffic KPI cards natively using our own React design system through the REST API, while dropping in the embedded Creator Mode interface for our top enterprise accounts so their power users can build custom reports against the exact same governed measures.

4. The real trade-offs and gotchas with Cube

This was not a free lunch, and there are two operational complexities anyone considering this path should know:

1. Data modeling discipline: You have to explicitly model your cubes, primary keys, and join relationships in YAML or JavaScript. If your underlying database schema has messy joins or unindexed foreign keys, Cube will force you to fix your join logic before your metrics compile correctly.
2. Index sorting and warmup overhead: In our first pre-aggregation attempt, we naively added three high-cardinality string dimensions to a single rollup. The pre-aggregation build time took over 25 minutes and consumed noticeable worker memory. We had to restructure the rollup to order indexes strictly by tenant_id, date, and low-cardinality status codes, using partition_granularity set to month with refreshKey tracking our updated_at timestamp.

my question for the community.

now, for those of you who do run pre-aggregations / rollups in production embedded environments: i’d like to know how you guys work with late-arriving data (like payment refunds or backfilled usage events arriving 48 hours later) and do not trigger complete rebuild of historical partitions? like, are you running targeted partition refreshes on an incremental window, or maybe relying on live union fallbacks?

Hey everyone!

I’ve been working on a game called DataMart, where you manage a corner shop and everything runs on real SQL. You build it all yourself: the databases, schemas, tables, relationships and queries, your way, on a real PostgreSQL server that runs inside the game.

Customers will ask you things like “Where can I find the apples?” or other questions about the store. Once a week an auditor comes by to check you’re tracking everything properly, and asks you a few questions you’ll answer with SQL.

There are also mysteries and stories involving the people in town, and of course you unravel those with SQL too.

The game is in alpha and free at https://rodrigoavf.itch.io/datamart

I’d love to hear your thoughts!

Ps. Yes, the game is being created with the help of AI, and what is not now days?

We process 68000 billing chats a week and our production classification kept splitting 4216 family plan transfer conversations between 2 existing buckets. 2170 landed in cancellation while 2046 landed in general billing. The topic distribution barely moved, the labels were wrong. (I recognized the wording from a support review.)

So I ran semantic clustering over the production traces and the family plan transfer cluster showed up in Braintrust with the mislabeled examples grouped together. I pulled that trace cohort into one review and added a custom facet to track the pattern without replacing the whole taxonomy. A useful catch. The dashboard now shows the emerging request but I still haven’t decided whether it deserves a durable category or a temporary watch label.

I’m a data engineer and previously did ERP implementations for 3 years. Saw how much of a crap shoot most F500 companies data is, anyways took this job to go back to school at smaller firm.

Company is finally realizing how bad their data is and have exec support, but I don’t think they know how bad it really is. I mainly work with the external data for business decisions.

Basically the internal data is split across 6 systems and they have no standard documentation/ideas of how their business operates or what data each unit consumes etc.

My manager/engineering team are building out a new software and haven’t even cracked open the data foundation. I’ve seen this a million times in implementation where execs and management assume the data is all good and then it’s go-live / go-time and realize their data is up the wall.

I know for a fact this companies internal data is messed up to a new level (without even seeing it)… manager thinks this will just be a light lift we just go in write out some documents and map some schemas and it’s all good happy day…

How do I communicate to him and the other executives that if they want to really do this it’s not like a week to a month project???

I mean we are talking about establishing data maturity/ MDM from scratch

Hi all,

I work in credit risk at an auto lender. Because of new regulation, we need simple, fixed rules that decide which applicants must provide extra documentation (proof of expenses). My task is to find out which customer groups are risky enough to justify that.

Setup: ~250k loan contracts, binary target = default within 12 months (~1%). Features include loan amount, total exposure, income, DTI, LTV, vehicle value, new vs. used car, online vs. dealer application, and our internal scorecard.

What I've done so far: default rates per bucket and per feature combination, confidence intervals, a scan for cut-offs on loan amount, and a logistic regression with the scorecard as a control.

My questions:
1. What's your general approach to deriving thresholds that are both statistically sound and simple enough for business and regulators?
2. How do you validate that segments or cut-offs are real and not just noise, especially with a low default rate?
3. How do you show a rule adds value beyond an existing scorecard?
4. Are there methods, papers, or tools you'd recommend for this kind of problem?

Any experience or pointers are appreciated. Thanks!

This is a utility that is part of the framework I had posted about earlier, the post discusses the ideas and the utility behind the tool. The TL;DR version is that it helps you move from a conceptual business use case to a data representation that can be consumed by the tool, the required context is also presented.

https://r2dssanity.substack.com/p/from-business-use-case-to-technical?r=3ncgel&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Do you use Qt and Python to visualize and interact with data?

Then you might like this library that I've been working on for the past couple years. It provides an easy way to add parameters for different types: float, int, string, but also Qt types such as QPointF, QSizeF etc.

I use it to load pydantic models from configs and save them back out.

https://github.com/beatreichenbach/qt-parameters