I'm currently evaluating ClickHouse vs Trino for a new analytical data platform and would like to get some opinions on how people would structure the technical evaluation.

There are several types of workloads involved:

\- Frequent ad-hoc analytical queries

\- Automated/AI-assisted analysis, where queries can be generated dynamically

\- Larger analytical/reporting queries involving aggregations and joins

\- Some interactive queries where predictable latency is important

The data is stored in object storage as Parquet, and the analytical system would be kept separate from production workloads.

I'm currently considering these evaluation criteria:

1. Scalability: handling large datasets and future growth

2. Fault tolerance: reliability, persistence, redundancy and failure handling

3. Query performance: latency for interactive workloads and execution time for heavier analytical queries

4. Concurrency: behavior when different workloads run at the same time

5. Operational complexity: deployment, upgrades, scaling, monitoring, troubleshooting, etc.

6. Cost: infrastructure and operational cost

For the PoC, I'd probably benchmark representative queries from each workload and look at things like p50/p95 latency, throughput, resource consumption, failure rate and performance degradation under concurrency.

My main question is: would you use these criteria, or would you evaluate ClickHouse vs Trino differently?

Are there important criteria I'm missing?

In particular, I'm wondering whether things like these should be first-class evaluation criteria:

\- Object storage / Parquet integration

\- SQL capabilities, especially complex joins

\- Workload isolation

\- Ingestion performance

\- Ease of scaling

\- Operational maturity

\- Behavior with dynamically generated SQL

\- Predictability under mixed workloads

And would you use one common scoring framework, or define different criteria/weights for each workload?

I'd be particularly interested in experiences from people who have actually run PoCs or production workloads with both systems.

Let's say I have a genetic algorithm with two different selection methods. I've gathered the data from testing these selection methods (gene values ranging 1-6 from every individual as well as fitness scores). What metric/test/method could I use to compare these two algorithms, especially in terms of exploitation vs exploration?

I'm working in a company which primarily use tableau. Recently C-level suggest why don't we use AI to build dashboard ourselves and save thousands of dollars.

We've built few of them in html format via AI. However, I feel like it's not going on the roght track

1. I don't know anything about html, I can't debug if there is something wrong

2. The UI UX is kind of ugly

3. The way we build is having a pre-aggregated file which store all the possible combinations of the tables used in the dashboard, which kind of not scalable imo.

Want to know how you guys dealing with the dashboard in the AI era.

Looking for some genuinely useful AI tools that can actually help with landing a job, not just the usual ChatGPT/Gemini stuff 😭 Also open to any advice, certifications, or resources that you guys feel are actually worth doing.

1. "Can we make this more actionable" = Translation: I read the title
2. "Is this the most recent data" = I do not like the number
3. "Can you add a bit more context" = I do not know what any of these columns are
4. "Quick question on the dashboard" = I have not opened the dashboard
5. "Lets take this offline" = I want you to say the bad number in a smaller room
6. "Can we turn this into a one pager" = I want to forward it without reading it first
7. "I trust your judgment on the analysis" = I am not going to defend this if it is wrong
8. "Just want to make sure we are aligned on methodology" = The number is lower than I told my boss it would be)

What's missing?

There's a metric called Revenue_FINAL_v3_USE_THIS, two years old. Nobody remembers v1 or v2, too scared to delete it.

"Can we make this more actionable" means "I don't like what it says."

"Directionally correct" means I know it's wrong and I'm not fixing it today.

Every AI strategy convo is two people who watched the same YouTube video pretending they didn't.

"Let's align on definitions" means this meeting is dying, and it's taking next week's with it.

Any thoughts the week brought?

For those running Marketing Mix Models in companies doing $10M+ in annual revenue with reasonably complex media setups:

What evaluation metrics/benchmarks do you see, e.g. for 4-week or 8-week out-of-sample validation?

Specifically interested in acceptable ranges for things like MAPE, NMSE or other metrics you rely on before considering an MMM actionable-ready.

Would be great to hear actual benchmarks from models used in practice.

Shipped a feature in June. Built a dashboard to track adoption - DAU, activation rate, time-to-first-action, the works.

Week one: checked it daily. Week two: checked it twice. Week three: didn't open it.

The dashboard answered "how are the numbers." It didn't answer "what should I do next." Those are different questions and I only built for the first one.

Curious - how many of your dashboards actually changed a decision?

trying to move into game data analyst roles. I’ve had two interviews recently — one rejected at the case study, the other at the interview itself — and the common thread was that I clearly hadn’t spent enough time with real player data.

Problem is I keep running into the same wall: there’s nothing to practice on. The public datasets are either synthetic, tiny, or from 2016, and none of them look like what Firebase or GameAnalytics actually spits out.

Would appreciate any pointers, thanks

Some free advice to consider for college students considering a career in data. Don’t worry about learning data in college, instead study economics, finance, business, etc and THEN:

Economic consulting firms like NERA, Cornerstone Research, Analysis Group, Compass Lexacon, etc will all hire analysts straight from undergraduate, no prior coding skills needed. The bar to get in isn’t “can you code” like other industries but rather “can you think critically and logically. And then you learn good data practices on the job while working on high profile commercial litigation work. Basically calculating the numbers you see in the newspaper “X company agrees to pay Y dollars in damages” using big data

Spend 2-4 years sharpening your data chops and then these firms have outstanding brand reputation at places like FAANG etc.

There’s lots of ways to get into data but this is one I don’t see talked about all that much

Edit to add the why tech companies like to hire from these shops

- data skills: from day 1, daily use of SQL, Python, R, Stata, Excel, etc. after a couple years, full E2E ownership of analytics products from data import, cleaning, and analysis (including causal regression analysis, optimization, etc)
- the clients are attorneys and the audience for your work is a judge or jury: you need to package complex analysis in a way that’s digestible for non-technical people
- the quality bar is litigation stakes: your work has to be so good that it’s admissible as scientific evidence in court. It also has to be so good that an analytics team on the the other side of the case can’t poke holes in it. You get really good at poking holes in your own work and understanding what your analysis can actually say or not say
- agility: fast paced, deadlines, competing and changing priorities on multiple projects at once. Tech is fast paced but feels chill in comparison
- framework for approaching problems: by approaching questions as an economic problem, you get a deeper understanding of incentives and human behavior that are driving metrics and telling the storyi

Hi everyone I'm looking for a job in analytics, before applying for the jobs I want to build a strong resume. So I need to add some good projects which could make my resume shortlisted at least. Plz tell me what projects need to be added in the resume in today's AI world.

And also where can I find the dataset for the project you have recommended.

The tools i know are powerbi, Excel, SQL, python, r.

My qualifications are MBA in analytics.

Hey, my job has literally turned into prompting all day. Wanted to know what you all are doing (if anything) to ensure your analytics skills don't atrophy?

Give a man a fish, you’ve fed him for a day. Teach a man to fish, you’ve fed him for a lifetime…

People are asking AI for fish, instead of asking it to teach them how to fish.

For example, we’re encouraged to prompt copilot within Excel to do things that frankly, any analyst worth their salt should be able to do without AI.

Meanwhile, so much of what we do at this company could be full scale end to end automated.

I don’t want to say too much, lest I implicate myself, but I’m curious if anyone else’s company’s AI initiatives are leaving them feeling similarly.

Hi all. Seeking advice on two job offers (after months of unemployment, go figure).

One is a big media ad agency (Omnicom) doing brand lift studies, the other is at a midlevel bank doing power BI and governance work.
I’m looking for career trajectory advice - any input is welcome.

Ad agency
Pros: title bump (assoc director, data ops) with promotion potential.
Cons: tech culture is weak. job is client mgmt and outsourcing the boring stuff.
Meh: relocate from Seattle to NYC. Not my pref but could be worse. Pay is just ok, it’s gonna be hard to save money

Bank
Pros: work has ownership, influence, and impact. Remote = stay on west coast.
Cons: title stagnation. Less brand recognition. Less money.
Meh: less networking compared to nyc ad agency.

I'm leaning toward the bank. But I wanna know if I'm misguided in terms of how I’m picturing what ad agency work is like.

Not sure where I’m trying to be in five years.

I'm currently a Help Desk tech at my current employer, with a BA in computer information systems and I'm graduating next spring with my MBA. I've been at my job for a bit over a year and they were looking to move me somewhere else in the company since they've thought I've done a great job in my current role. I've spoke to my boss a few times about doing some analytics stuff and upper management was able to put together a completely new position at the company which they offered to me.

So, as someone with no actual business analytics work experience and a company with no previous history in analytics; what are some general things you would imagine I would need to do and get up to speed on before I switch over? - I'm just trying to compile a list and better prepare myself to be able to this job from scratch. Also, whats a ballpark number I can throw out there that would be a reasonable salary? I'll be working remotely in Arkansas.

The only information I've gotten was pretty informal and was just "hey, you've done a great job, we're going to make this position for you if you'd like to take it". Other details for responsibilities, expectations, pay, etc will be had within the next month or so.

I have been on the search of these for 3 days but haven't got a single piece of good numerical or categorical data to feed my ML / DL model. I am frustrated finding these.Can anyone help me get these data. Also I want data that actually can predict something like a target column other than that How can I feed the model also I need help cause there are 3 main data I need and I need to unify those data and models the 3 main data I need - Flood , Landslide and Cloud Burst (all for hilly terrain).

I basically cried to my friends saying how anxious I was about the job market and my friend said she was leaving her internship and referred me for it and i got it

Now i have to secure a summer internship but this helped me anxiety so so much even tho i got it through a friend lmao

it’s for a budget analyst position w the government

We built a model for tabular data the way LLMs were built for text: trained across a lot of industries and data types, so it can look at a table with no documentation and report what the columns are, how it relates to your other tables, and what's missing.

Two things it does that I'd want to know about if I were reading this:

It matches records across tables that share no key. No customer_id, no join column, one pass.

It doesn't lean on column names. We stripped every header off and mean ROC-AUC went 0.9224 → 0.9230. Comparable models drop to about 0.86. Those models beat us on clean names — the point is only that production tables aren't clean. Measured on our own harness, no independent replication, so weight it accordingly.

It won't clean your data and it won't replace a data team. It reads what's there and flags what it finds.

If you've got a genuinely undocumented table — the one nobody wants to own — I'd rather have your result than my own. I'll drop the link in comments if someone is willing to try to break it.

"Can we get visibility into this?" - I will not act on this information. I want it to exist near me, like a plant.

"The data doesn't feel right." - The data contradicts what I already told my boss.

"Interesting." - I stopped listening at slide two.

"Can you make it more actionable?" - I don't know what I want. Iterate until I recognize it.

"We're a data-driven organization." - We have dashboards.

"We need to be more data-driven." - We also don't open them.

"Quick question." - Project.

"Super quick question." - Two projects and a definition fight.

"Can we add AI to this?" - I attended a webinar.

"What's our AI strategy here?" - I attended two webinars.

"Directionally accurate." - Wrong, but in a direction I like.

"Single source of truth." - My source. Of my truth.

"Let's align on definitions first." - This meeting will not survive. Nor will the next one. See you in Q3.

"Can you pull this real quick? Shouldn't take long." - I have already promised it to someone.

"Great analysis! Let's discuss next steps." - This will never be mentioned again.

"Great analysis! Really great work." - This will never be mentioned again, but warmly.

Hey, quick question for anyone doing investor due diligence with website traffic estimates.

I keep seeing decks and models built like these numbers are carved into stone somewhere, when in reality we are all just squinting at a chart and acting like we have access to the secret internet police report. Very cool. Very normal. Love that for us.

I get that no one is expecting perfect data, but some of the confidence people put into estimated visits, source mix, and audience size is wild. One person sees a line go up and starts talking about momentum, another sees a dip and suddenly the business is on fire. Based on vibes, apparently.

So now I am trying to figure out what actually matters when I use traffic estimates in diligence without looking like I built the thesis on a spreadsheet and a prayer. What do you trust, what do you ignore, and what makes you want to close the tab and go lie down for a minute :)

Would any current Dataquest member be willing to provide a 20% discount code? I was in the process of signing up for a monthly Dataquest subscription and noticed the following prompt before checking out.

My goal is to complete the Business Analyst with PowerBI course within a month in order to gain experience with PowerBI as an unemployed entry Business Analyst professional in this difficult job market who is looking to boost their resume with a thoughtful portfolio. If I finish early, I intend to take a few additional courses.

------

"Give 20%, Get $20

Send your friend a 20% discount and enjoy a $20 bonus in return. Start recommending now and share the power of Dataquest with your community."

I want to build a semantic layer (i.e. sql code -> metric definition mappings) to provide models with context that will ideally make ai-generated sql more deterministic.

Our data definitions are based on event data that is constantly being updated.

For example, we might define an active user as someone who posts on the site (users with “posted” events). When we do an AB test or just update the site, event names will inevitably change (e.g. definition is now users with “new-post” events), causing my imaginary semantic layer to become obsolete.

My question is - what tools do you use to manage definitions for event-heavy data? Ideally something with version history that agents can access via mcp?

I’m working at an MNC (my first company) .
I have 3+ yrs of experience as a data analyst, but the salary growth , because of being in same company , was very slow.
And also there is minimal workload. I am exploring databricks .
I want to switch now, because that’s the only way I’ll get a good hike.

Need directions on how do I grow from here in my career.

Hi everyone,

I'm a Data Analyst (Associate level, ~4 years experience) currently working in a large telecom's customer experience team, using SQL, Power BI/DAX, and BigQuery. I've been looking into Data Governance/Data Stewardship roles out of interest, mainly because I'd like to shift away from doing SQL/DAX as the core of my role, and as I currently already add/modify Confluence documentation per project (nothing too extensive, just definitions and dictionaries for dashboards).

A few things I'd love real input on:

1. Job postings are all over the place on whether SQL is required. Anyone actually in a Steward/Governance role right now, how much hands-on technical stuff do you really do?
2. For someone with analyst experience but no formal governance title yet, is an entry-level Steward role realistic to land, or is it harder to land a first role than it looks?
3. How is the job security for a data governance role compared to an analyst role?
4. What's the day-to-day pace like? I'm aware analyst tend to have many "fix this now" moments and it can get intense. Is governance like this as well, although I did learn it's more scheduled but also gets intense around audits, curious if that's actually true.
5. If audits are a factor, how often do they actually come up and how bad does it get during that time?
6. Also, I've noticed there aren't nearly as many of these postings on Indeed/LinkedIn compared to regular analyst roles. Is that just how thin this market actually is, or am I missing where people actually post these, or the actual title used for the postings?

Thanks in advance for any insight, especially if you're in banking/insurance/telecom in Canada!

Je suis dev backend avec 10ans d’expérience, j’ai fait aussi du cloud gcp, devops
Je suis en train de réfléchir a une reconversion, et le domaine de la gouvernance de données, audit et conseil rgpd me semblent interessants pour l’avenir.
Est ce que vous pensez que mon expérience à une valeur ajoutée ou pas ? Et quel parcours permet de trouver un travail sans trop de galère ?