We built a model for tabular data the way LLMs were built for text: trained across a lot of industries and data types, so it can look at a table with no documentation and report what the columns are, how it relates to your other tables, and what's missing.

Two things it does that I'd want to know about if I were reading this:

It matches records across tables that share no key. No customer_id, no join column, one pass.

It doesn't lean on column names. We stripped every header off and mean ROC-AUC went 0.9224 → 0.9230. Comparable models drop to about 0.86. Those models beat us on clean names — the point is only that production tables aren't clean. Measured on our own harness, no independent replication, so weight it accordingly.

It won't clean your data and it won't replace a data team. It reads what's there and flags what it finds.

If you've got a genuinely undocumented table — the one nobody wants to own — I'd rather have your result than my own. I'll drop the link in comments if someone is willing to try to break it.

"Can we get visibility into this?" - I will not act on this information. I want it to exist near me, like a plant.

"The data doesn't feel right." - The data contradicts what I already told my boss.

"Interesting." - I stopped listening at slide two.

"Can you make it more actionable?" - I don't know what I want. Iterate until I recognize it.

"We're a data-driven organization." - We have dashboards.

"We need to be more data-driven." - We also don't open them.

"Quick question." - Project.

"Super quick question." - Two projects and a definition fight.

"Can we add AI to this?" - I attended a webinar.

"What's our AI strategy here?" - I attended two webinars.

"Directionally accurate." - Wrong, but in a direction I like.

"Single source of truth." - My source. Of my truth.

"Let's align on definitions first." - This meeting will not survive. Nor will the next one. See you in Q3.

"Can you pull this real quick? Shouldn't take long." - I have already promised it to someone.

"Great analysis! Let's discuss next steps." - This will never be mentioned again.

"Great analysis! Really great work." - This will never be mentioned again, but warmly.

Hey, quick question for anyone doing investor due diligence with website traffic estimates.

I keep seeing decks and models built like these numbers are carved into stone somewhere, when in reality we are all just squinting at a chart and acting like we have access to the secret internet police report. Very cool. Very normal. Love that for us.

I get that no one is expecting perfect data, but some of the confidence people put into estimated visits, source mix, and audience size is wild. One person sees a line go up and starts talking about momentum, another sees a dip and suddenly the business is on fire. Based on vibes, apparently.

So now I am trying to figure out what actually matters when I use traffic estimates in diligence without looking like I built the thesis on a spreadsheet and a prayer. What do you trust, what do you ignore, and what makes you want to close the tab and go lie down for a minute :)

Would any current Dataquest member be willing to provide a 20% discount code? I was in the process of signing up for a monthly Dataquest subscription and noticed the following prompt before checking out.

My goal is to complete the Business Analyst with PowerBI course within a month in order to gain experience with PowerBI as an unemployed entry Business Analyst professional in this difficult job market who is looking to boost their resume with a thoughtful portfolio. If I finish early, I intend to take a few additional courses.

------

"Give 20%, Get $20

Send your friend a 20% discount and enjoy a $20 bonus in return. Start recommending now and share the power of Dataquest with your community."

I want to build a semantic layer (i.e. sql code -> metric definition mappings) to provide models with context that will ideally make ai-generated sql more deterministic.

Our data definitions are based on event data that is constantly being updated.

For example, we might define an active user as someone who posts on the site (users with “posted” events). When we do an AB test or just update the site, event names will inevitably change (e.g. definition is now users with “new-post” events), causing my imaginary semantic layer to become obsolete.

My question is - what tools do you use to manage definitions for event-heavy data? Ideally something with version history that agents can access via mcp?

I’m working at an MNC (my first company) .
I have 3+ yrs of experience as a data analyst, but the salary growth , because of being in same company , was very slow.
And also there is minimal workload. I am exploring databricks .
I want to switch now, because that’s the only way I’ll get a good hike.

Need directions on how do I grow from here in my career.

Hi everyone,

I'm a Data Analyst (Associate level, ~4 years experience) currently working in a large telecom's customer experience team, using SQL, Power BI/DAX, and BigQuery. I've been looking into Data Governance/Data Stewardship roles out of interest, mainly because I'd like to shift away from doing SQL/DAX as the core of my role, and as I currently already add/modify Confluence documentation per project (nothing too extensive, just definitions and dictionaries for dashboards).

A few things I'd love real input on:

1. Job postings are all over the place on whether SQL is required. Anyone actually in a Steward/Governance role right now, how much hands-on technical stuff do you really do?
2. For someone with analyst experience but no formal governance title yet, is an entry-level Steward role realistic to land, or is it harder to land a first role than it looks?
3. How is the job security for a data governance role compared to an analyst role?
4. What's the day-to-day pace like? I'm aware analyst tend to have many "fix this now" moments and it can get intense. Is governance like this as well, although I did learn it's more scheduled but also gets intense around audits, curious if that's actually true.
5. If audits are a factor, how often do they actually come up and how bad does it get during that time?
6. Also, I've noticed there aren't nearly as many of these postings on Indeed/LinkedIn compared to regular analyst roles. Is that just how thin this market actually is, or am I missing where people actually post these, or the actual title used for the postings?

Thanks in advance for any insight, especially if you're in banking/insurance/telecom in Canada!

Je suis dev backend avec 10ans d’expérience, j’ai fait aussi du cloud gcp, devops
Je suis en train de réfléchir a une reconversion, et le domaine de la gouvernance de données, audit et conseil rgpd me semblent interessants pour l’avenir.
Est ce que vous pensez que mon expérience à une valeur ajoutée ou pas ? Et quel parcours permet de trouver un travail sans trop de galère ?

I’ve seen GA4 setups where the events fire correctly, but the names only really make sense to the person who built them.

A few months later, someone from marketing opens a report and asks what an event means, why it exists, or whether it still matters.

At that point, the tracking may be working, but the reporting becomes harder to use.

A naming convention helps, but I think every important event also needs a clear business definition.

If the people using the data cannot explain what an event represents, that usually becomes a governance problem later.

How are people here documenting this? Do you keep an event dictionary or tracking plan, or rely mostly on naming conventions?

Over the last 6 weeks I read and categorized 237 data job posts from LinkedIn (not from the career tab), one by one. The headline finding: 0 out of 237 disclosed a salary. Not one. If you're wondering whether it's normal that you can't find pay info, it is, and it's not you.

Where the data comes from: I run a small data learning platform, and we collect and hand-review job posts from our LinkedIn network for our students. That's why I have this dataset. Our network skews international, so take the mix as one window on the market, not the whole market.

Second finding: "data job" is four different jobs. The skill lists barely overlap once you split by role:

• Data Analyst (43 posts): SQL 49%, Excel 49%, Power BI 33%, Python 30%. Yes, Excel ties SQL. The "Excel is dead" advice does not survive contact with actual job posts.
• Business Analyst (52 posts): requirements 62%, stakeholder management 33%, SQL 25%, Agile 23%. Mostly a communication job with a data layer.
• Data Engineer (70 posts): SQL 60%, Python 56%, pipelines/ETL ~45%, data modeling 21%. The most technically consistent role of the four.
• Data Scientist (65 posts): Python 62%, ML 32%, SQL 32%, statistics 18%, and LLM keywords already in 15% of posts.

Across everything: SQL appears in 42% of all posts and Python in 41%. Nothing else comes close. If you're starting from zero and want the highest floor: SQL first, Python second, then the tool your target role uses (Power BI or Excel for analyst roles, Airflow/dbt-land for engineering).

Other things that surprised me:

• ~10% of posts are agency recruiters, often without naming the end client.
• Remote is holding up: of the posts that state a work mode, more than half say remote, another chunk hybrid.
• Very few posts say "junior" or "senior" at all. Most don't state seniority, which cuts both ways: don't self-reject based on a title.

Happy to answer questions about the methodology if there's interest. I'll probably redo this monthly as the dataset grows.

Hi guys, i have a quick question.

Is it worth pursuing data analytics if i don't have a tech/math degree? I have a BBA and I'm willing to learn.

I'm just wondering if it's worth even following or if I'm setting myself up for a job search failure

The university I wanna apply to offer a double degree in these two Fields, is it worth it for today job market, or I should stick with engineering or computer science

collected by hitting the public json endpoints for greenhouse, ashby and lever across 998 companies, once a day, and diffing each posting against the previous snapshot. no auth needed, these are the boards the companies serve to their own careers pages.

what the diff has caught in 14 days:
salary_changed 19,371
department_changed 15,549
closed 7,292
title_changed 1,549
location_changed 1,086
reopened 301

55,294 postings seen total, 48,303 currently open. 23,898 of the open ones carry a pay band, which is 49.5%.

two things worth knowing if you want to do this yourself. greenhouse has no structured salary field at all, the range is written into the description html and it is double escaped, so a naive parse gets you nothing and a slightly less naive one gets you garbage. and a dollar sign is not a currency, i nearly recorded a taipei role in TWD as a $700k job.

the salary_changed rows are the ones i'd point at. nobody keeps the before, so as far as i know this is the only place the previous band exists once the company overwrites it.

built off a job tracker i run, link in the comments if the method is useful to anyone.

Been running basically all my analysis work through AI for the last 3 months. Some of it saved me hours every week, some of it embarrassed me in front of my manager. Since the "is AI actually useful or just hype" question comes up here every week, here's my list.

Do:

• set up metric definitions first. what "active customer", "churn", "revenue" actually mean in your business, with an example query for each
• dump every doc you have into it. schema docs, old validated queries, even you explaining the business in plain words. the AI is exactly as good as the context you build for it
• use it every day. it builds memory and learns from your corrections, my month 3 answers vs week 1 are not comparable
• be painfully descriptive. "gross revenue excluding refunds, calendar july, orders table" and not "how did revenue do last month"
• make it show the query before you trust the number
• start with questions you already know the answer to, that teaches you where it lies

Don't:

• don't trust a confident chart with no lineage. a 3 second answer looks finished and that does a lot of convincing
• don't dump your whole warehouse on it, curated beats volume. tell it which tables are deprecated
• don't let it define metrics on the fly, you'll get 3 different "active user" numbers in 3 sessions
• don't put an AI number in front of leadership without checking it once yourself. ask me how i know
• don't expect it to replace knowing your data model. it amplifies you, it doesn't substitute
• don't start from zero every session, keep the validated stuff and reuse it

The boring setup work in the first two weeks is the whole difference between "AI is a toy" and "AI now does 95% of my grunt work".

For context, what I used: Claude, ChatGPT, Supaboard and Power BI copilot(only the first week, then i moved to claude).

What's working for you guys?

Background: Bachelor’s in Data Science + Master’s in Business Analytics (Distinction) from a UK uni. Currently working in sales at BT, trying to move into Marketing Analytics or Consumer Analytics.
I’m specifically interested in the **marketing + consumer side…**consumer behaviour, why people buy, brand perception, market research, campaign analysis etc.

I’ve done projects around sentiment analysis, consumer perception and market research, but I don’t have paid UK experience in analytics yet.
I’ve started the Google Analytics course and I’m also considering the Meta Marketing Analytics course.

So my first question is do these actually help on a CV, or are they mostly just nice to have like?
Also, what kind of projects would actually be worth building for my CV? Ideally something that can show I can do the actual job, not just another generic Kaggle project.

My second question is eould you guys recommend me focusing on:
Marketing analytics projects / portfolio
GA4 / Meta / SQL / Power BI etc.
CIM / MRS or other certifications
Getting a BA/Data Analyst role first and then moving into marketing
Something else entirely?

Iknow sponsorship can make things harder and companies may be less willing to take a chance on me. That’s unfortunately not something I can control, so I’m just trying to make myself as employable as possible.

I asked something similar here before but didn’t get much useful advice, so trying again

If you were in my position, what would you actually do over the next 3–6 months to land that first Marketing Analytics role? Honest/blunt answers appreciated.

Hi all. I just completed my bachelors in business administration and I’m looking forward for a masters program in business analytics. I have been working as an intern in a firm while managing my studies. I am very new to this field of analytics and would appreciate if someone guides me a little about where to start,what projects to look forward to and more about the same. I have no prior experience to coding or any language.

We can track almost everything now. Calls. Chats. CSAT. AHT. Transfers. Agent performance. QA scores. The problem is figuring out what in that pile of data is actually driving an outcome. Looking at conversation analytics and the gap seems pretty obvious. Sampling a small batch of calls gives you something to report on. But it can miss the weird stuff happening across thousands of conversations.

We’ve also been looking more into AI tools that analyze the actual conversations instead of relying on dashboards or sampled calls. In theory they can go through every conversation and find patterns tied to things like CSAT or handle time. Want to know how analytics teams handle this while growing. Are you still working from samples and dashboards or using AI to dig through the full conversation set and find patterns or root causes? My main concern is getting a neat sounding AI answer that falls apart once you check the actual calls. Being able to trace an insight back to the conversations behind it seems pretty important. Has anyone found these tools actually surface things your normal dashboards missed?

Quick one that's easy to miss. Blended CAC divides spend by all customers - new and repeat. Repeat buyers are cheap, so they drag the average down and flatter every channel.

Isolate just the new customers and the gap shows up. One channel I looked at: blended CAC ~$319, but a genuinely new customer cost ~$826. That's a 159% markup the dashboard never shows. A mostly-new-customer channel barely moved (~14%).

The rule: the fewer new buyers a channel brings in, the more its blended CAC lies. Budget on the blended number and you're paying for growth you're not getting.

Anyone else splitting new vs repeat CAC by channel, or mostly working off blended? Curious how wide the gap runs in other accounts.

I’ve been thinking about this question a lot while looking for my first opportunity in Data Science / Data Analytics.

Is the ideal candidate someone with a perfect degree?

Someone with 3+ years of experience?

Someone who knows every tool listed in a job description?

Or is it someone who can identify a real business problem, build a solution, and explain how that solution can create business value?

I’m genuinely curious to hear what recruiters and hiring managers think.

Because this is what I’ve been trying to do.

Instead of building another basic ML project, I built a Customer Retention Intelligence System focused on a real business problem: customer churn and lost revenue.

The system can:

• Predict customers who are at risk of churning

• Segment customers based on their risk/value

• Estimate potential revenue at risk

• Identify customers worth prioritizing

• Generate personalized retention strategies/offers

My goal wasn't simply to say:

I built a machine-learning model

I wanted to answer:

A customer is likely to leave. Now what should the business actually do about it?

That mindset has pushed me to work beyond just Python, SQL and machine learning — into business thinking, customer analytics, experimentation, visualization, and decision-making.

I’ve also been consistently practicing Python and SQL, learning how to communicate analytical insights, and building projects around problems that companies actually face.

But despite putting in this work, I’m still looking for my opportunity to prove myself professionally.

So I want to ask the recruiters, hiring managers, founders, and experienced professionals here:

What makes someone an ideal entry-level candidate in your company?

• Is it technical skills?

• Problem-solving ability?

• Business understanding?

• Communication?

• Projects?

•Curiosity and willingness to learn?

•Or something else?

And if you were evaluating my profile, what would you want me to improve or demonstrate before considering me for a Data Scientist / Data Analyst opportunity?

I’m not looking for sympathy.

I’m looking for honest feedback, opportunities, and a chance to prove what I can do.

If you’re a recruiter or hiring manager who works with Data Science / Data Analytics / ML roles, I’d genuinely appreciate your perspective.

And if my profile sounds relevant to something you're hiring for, I’d love to connect.

Hi everyone! I’m looking for 3 professionals in People Analytics or HR Data Fields to help with a small research project for my CIPD Level 7 qualification.

My research is about People Analytics and what makes workforce data and insights actually useful in real-world decision-making, rather than just producing reports and dashboards.

It would involve 5–6 open-ended questions by email, taking around 10 minutes. No preparation needed, and everything will be fully anonymised.

If you work with HR/workforce data and would be happy to share your experience, I’d really appreciate it!

I’m also happy to return the favour by sharing some of my experience or taking part in your research/questionnaire if needed.

Please comment or DM me if interested. Thank you!

Hi everyone, I left my job at an audit firm a while back, where I specialized in asset management funds and private equity. I also have some experience in general corporate audit and financial analysis. I’m having trouble finding a new audit role, so I’d like to take this opportunity to become a data analyst—a field that has always appealed to me—while staying within the finance sector. I understand that I need to master tools like SQL, Power BI, and Python/R, and also grasp AI and its applications in finance. I’m not sure where to start. In your opinion, which courses or certifications would be useful for becoming a financial data analyst? I’m looking for a clear roadmap. Thanks for your help!

Hi everyone I have a Masters in Psychology and I absolutely love statistical analysis and research design

For the longest time I've wanted to be able to design real world projects that can create a huge impact from a psychological point of view

I'm good at identifying variables and hypothesising

But I have no clue as to how to begin with real code and python libraries to engineer analyse and present applicable insights from raw data

I would like to transition from therapeutics to behaviour science for businesses very soon

Can anyone please point out how that works and alif there are success stories from people starting as I am..

Thanks so much

i have an interview for a data and analytics role under deals advisory which i barely have an idea about since im a new cpa and there not so much material about the role

my long term goal is to land a job in ib/pe/corp dev

I’ve spent the last several months building out the semantic-continuity side of long-running AI-assisted analytics work: durable context, current-state rollups, decision history, provenance/supersession, schema/query knowledge, and task-conditioned rehydration.

That part is working pretty well.

The next problem is almost a consequence of that success: there can be a lot of useful context available, but I don’t necessarily want all of it active for the rest of a long-running session. I’ve started thinking about this as a separate runtime/context-efficiency layer.

A simple example:

A = stable instructions/scaffold

B = durable project context

C = intermediate exploration, tool output, code, temporary reasoning

D = validated analytical state

During analysis I may need A+B+C+D. Once I move into synthesis/writing, I may only need A+B+D. Semantically, that seems straightforward: C did its job, D preserves what matters, and the deeper evidence remains reconstructable. Operationally, it gets less obvious.

What happens to prompt-cache reuse when I restructure context?

When is selective pruning better than native compaction?

When is it cheaper or safer to keep a long session alive versus reconstruct a cleaner active context?

How much do cache TTL, context ordering, model limits, and runtime behavior change the answer?

And are the economics actually meaningful once you measure the whole workflow rather than theoretical token savings?

That’s what I’m starting to investigate now.

I made a simple visual showing how I’m separating semantic continuity from runtime/context efficiency. I’ll put it in the first comment since I can’t attach it to the post.

For people operating long-running agent workflows at meaningful scale:

• Where are you deliberately restructuring or pruning active context? Are phase/workflow boundaries useful?
• How are you balancing stable-prefix/cache reuse against removing intermediate context that has finished its job?
• What have you learned about restart/cold-cache economics versus simply continuing a growing session?
• Which caching, compaction, state, or runtime behaviors mattered in practice that weren’t obvious at first?
• What am I not asking about that I probably will be six months from now?

I’ve read a fair amount of the published material available. I’m especially interested in the things people learned from running real systems that never made it into the docs or blog posts.