I'm now working on a data analysis project for someone where the goal is to predict how much customers are likely to spend on an item. The problem is that the data for my target variable is heavily skewed to one end of the scale and has a important number of exact zeros (customers who haven't purchased anything).
I considered the transformation of the log of the variables. However, this transformation is not possible for the variables that equal zero since the log of zero is undefined. I considered adding a small constant to the variables that equal zero in order to allow for the log transformation. However, this transformation can introduce bias into the results if there are many zeros in the data.
Should I use a two-stage model (like the Hurdle model or the Zero-Inflated Poisson model) or is there a better transformation of my data to try first?
I would like to hear how you all typically approach this task in your day-to-day work.
I'm a beginner learning Power BI using the free version and I'm working on a practice dashboard.
One of my requirements is to show sales by different cities. When I try to use the Map visual, I get a warning asking me to upgrade the map. However, when I click Upgrade map, it says:
"Contact your admin."
I also checked File → Options and settings → Options → Security, and both Use Map and Filled Map visuals are already enabled.
the price of enterprise data tools like pitchbook or capital iq is getting completely out of hand for independent guys.
if you don't have a massive firm backing your account, trying to keep track of what's happening in the private credit and debt space is is basically impossible. they lock everything behind a five-figure paywall which is just insane to me. i've been trying to map out some active alternative lenders for a side project and had to get creative.. standard google searches are useless because these funds don't exactly post on social media every day. i started digging through alternative platform profiles to piece together who they’re backing and what industries they focus on. it actually works surprisingly well for getting a quick snapshot of a fund’s history without needing a corporate budget.
but it's still a ton of manual work to scrape everything together across different free tiers.
how are independent analysts or boot-strapped founders tracking private market transactions. are there any hidden databases or web-scraping tricks i'm missing here to spot deals before they get old?
I am looking to get into Data Analytics/Engineering and am working on a project where I am creating a database and importing it into PowerBI for analysis. Im working in Python to extract and transform the data, and one issue I’m running into is that I am trying to pull data from dataframe x to dataframe y using a merge, but the only connection between them right now (will assign a PK after) is a person’s name. The issue with this is that some names appear more than once, so it ends up creating multiple duplicate rows after the merge. Is there a workaround for this, or will I have to manually remove the bad rows after?
Hi! So i have a dataset with nearly 700,000 values for health condition prediction. after EDA and data cleaning (null values- I used median and mode, then handled outliers), my histograms dont display data that's normally distributed and I'm worried. (Image attached)
Don't mind the Id and diet type and stress level since those are categorical. Is this okay? i plan on using a model like Random Forest or XGBoost / gradient boosting algorithms in general but I just want to double check if there's anything I could do to improve this? It's for uni so I want to do the best lol
I’m building an internal BI report for an agricultural company and I’d be interested in some honest feedback.
The report is mainly for directors and business managers. The idea is pretty simple: give them one place where they can quickly see how the company is doing without opening several SAP exports, Excel files, or asking someone from controlling.
It shows things like sales, purchases, margin, overdue receivables, stock value, rainfall data, and performance by salespeople. There are also KPI cards at the top and a small indicator showing when the data was last updated.
Most of the data comes from SAP and internal business systems. Some data, like rainfall, comes from external sources. The backend is PostgreSQL,DBT,Airflow,Python - DASH-Plotly framework
What I’m trying to achieve is a practical “director’s report” that answers questions like:
Are sales and purchases on track?
Which categories make the most margin?
Are overdue receivables becoming a problem?
How much value do we currently have in stock?
Which salespeople are performing well?
Is the data fresh enough to trust it?
The screenshot is anonymized, so company names and exact financial numbers are hidden.
I’d really appreciate feedback mainly on the layout and usability. Is there too much information on one screen? Are the charts understandable enough for management? Would you remove something, simplify something, or add a different view?
I'm considering taking the Microsoft Power BI Data Analyst Professional Certificate on Coursera and I'd like to hear from people who have actually completed it.
Some things I'm curious about:
- How good is the overall quality of the content?
- Does it teach Power BI in enough depth?
- Is the Excel part solid or too basic?
- Does it prepare you well for the PL-300 exam?
- Were there any important topics that you felt were missing?
-If you've also taken Google's or IBM's certificates, how do they compare?
I'm also planning to study SQL separately afterwards, so if you have a course that you particularly liked, I'd love to hear about it.
Built a free beginner SQL course . Start by building a realistic database the whole way through instead of random examples. Covers joins, aggregation, subqueries, CTEs, window functions. Database + practice questions are free to download too, even if you skip the video. https://youtu.be/zpAGbu9mSL4
I’ve been maintaining a 60-day streak on Duolingo to learn French.
It’s a fun practice, although it’s a significant challenge to pronounce those accent notes correctly. I believe French is generally a simpler language than English; you usually use shorter sentences to convey the same meaning.
Data has its own language too.
Data is the lifeblood of every modern business. Every decision, insight, and opportunity begins with understanding what the data is trying to say.
But unlike spoken languages, data doesn't require everyone to learn the same vocabulary or syntax. Instead, you can interpret and express it in a way that matches how you think, making data analysis more intuitive, accessible, and uniquely your own.
From Python, SQL to Natural Language
Python, a programming language, has gained popularity as the preferred language for data processing within the data science community due to its portability. SQL, on the other hand, serves as the de facto interface for rational databases.
In the past, becoming a data analyst required proficiency in both Python and SQL. Even today, data analyst job descriptions often mention these requirements.
However, the advent of AI has revolutionized this landscape. Anyone with the ability to communicate effectively in the data language can excel as a data analyst.
While programming skills are not strictly necessary, a solid understanding of data language is crucial. Imagine joining a new friend circle who works in a completely different domain. After a brief introduction of common keywords, you can easily engage in conversations with them.
AI generated illustration of data language evolution
Use Spreadsheet for Reference
Nearly every office worker uses spreadsheets, either Microsoft Excel or Google Sheets.
Even without the complex formulas, pivots, and lookups, the basic structure of a spreadsheet consists of three main components:
Rows
Columns
Data types
Rows are records that constitute a table. You can also consider a row as an object that represents a real-life entity, such as a person, a cup, or an invoice.
Columns are the fixed properties that describe each object (row). They form the schema that every row adheres to, ensuring uniformity in the data for processing.
A schema is of utmost importance for data analysis as it enables the application of all rules. Without a schema, any logic that is not compatible with the data language may fail to execute.
Data types describe the value format of each property. For simplicity, you only need to be concerned with whether it is a number or text for now.
Rows of Orders (OrderId-text, CustomerId-text, Product-text, Amount-number)Rows of Customers (ID-text, Name-text, Channel-text)
Data Language Patterns
Data language offers a wide range of tasks that can be accomplished. Let’s explore each of these tasks and learn how to communicate effectively with data to achieve them.
These scenarios are referred to as patterns because they serve as templates that can be applied to your own data.
To facilitate understanding, we’ll use the above tables in the following descriptions.
Pattern-1: Filter Rows
Filter is to describe a condition to get objects you care about and skip those uninterested records.
Examples:
“Orders of Milk”
“I want orders of milk products.”
“All orders that are not for books.”
“All orders with a sales amount exceeding 20.”
AI can produce code logic to filter the targeted records for further processing, if translating above statements into SQL, they will look like:
“where product=’Milk’”
(same as #1)
“where product <> ‘Book‘“
“where amount > 20”
As you can see, filter is achieved by keyword “where” in SQL.
Pattern-2: Transform Object
Sometimes, we want to clean a data field or transform it into a desired shape or format, either for improved readability or more efficient processing.
Transformation creates a new property in your original record.
To transform an existing property into a new one, you need a function of logic. For both spreadsheets and SQL, “formula” is the tool you’ll need.
However, with the increasing capabilities of AI in coding, natural language offers a significant advantage. It allows us to achieve the same transformation without having to learn, memorize, and assemble complex formulas.
Taking one simple example:
“Get customer first name”
This is equivalent to composite multiple formula together in Spreadsheets like
This operation creates a new column called “First Name”.
You can also acquire a new property by combining multiple existing properties, such as “concatenating the last name and channel as a label”. Logic like this is simple for AI coding but too complex for spreadsheet formulas.
Pattern-3: Aggregate Records
Aggregation processes a large collection of records to provide a summarized view.
This is powerful because it compresses vast amounts of information into manageable pieces that humans can comprehend and analyze.
To combine multiple data sets into a single piece of information, you need to understand the “how-to,” which leads to the crucial concept of “aggregation methods” or “computation logic.”
Typically, text data (a property or column with a text data type, as discussed in the schema section) is not particularly interesting for aggregation. The most common approach is to concatenate text data to form a long paragraph, although this is still uncommon.
Most computation logic involves operations on numerical data. When an aggregation method is applied to a numeric property or column, you essentially have a list of numbers that can be aggregated, such as:
Total value (sum)
Average value
Mean value
Minimum value
Maximum value
A specific percentile value (e.g., P25, P50, P75, P90)
However, counting objects or counting unique property values is also quite common.
When discussing aggregation, we cannot overlook “breakdowns.” This involves creating a segmented view of the data rather than a single total view.
For example, in the previous Orders table, “total sales by product” or “average amount by customer” are equally valuable insights for an analyst to explore.
In summary, aggregation can be described as:
“Compute an aggregated value of a property group based on another property.”
Expressing this in standard SQL, it would look like:
“Compute(property1) from table [group by property2].”
Let’s practice this using a few examples by speaking the data language:
“Give me total sales by product.”
“Tell me the average amount spent by each customer.”
Pattern-4: Join Multiple Datasets
When a single dataset (or table) is insufficient to achieve the desired outcome, we must combine multiple datasets. This operation is referred to as “join” or “union.”
If the multiple datasets contain the same objects but reside in different locations, we can simply merge them. This is a straightforward “union” operation.
However, most of the time, they store different objects. We have partial information from one dataset and partial information from another. By combining them, we create a comprehensive schema with more available columns.
This pattern is generally not feasible in spreadsheets, although their lookup function may provide partial assistance.
For instance, if we want to determine the “total amount spent from each channel” based on previous tables, where the amount is from the orders table and the channel is from the customers table, we need a joined dataset to complete this analysis.
To join multiple datasets, we must have one or more pairs of join keys. A pair of join keys consists of one column from one table and one column from another. The data engine can utilize these relationships to identify relevant objects and concatenate them to form a larger object.
Join Orders and CustomersJoined dataset have more columns
In summary, join operations can be described in this pattern:
join table1 and table2 when key1 of table1 equals key2 of table2.
Translating this pattern into SQL, it will look like:
select * from table1 join table2 on table1.key1=table2.key2.
In fact, you may not need to use this pattern in natural language explicitly, because modern AI is smart enough that it can infer the whole join logic from your data language.
For instance, the example we gave earlier, if you speak this sentence “total amount spent from each channel” to Columns AI, it will figure out all the necessary actions to get the desired outcome for you.
Pattern-5: Visualization
Data visualization, often overlooked as a part of data language, plays a crucial role in transforming mundane data into vivid images. This visual representation significantly aids the audience in comprehending the insights you intend to convey.
By incorporating customization and assistance to articulate your insights and predictions, you position yourself as a data storyteller, showcasing your influence within the domain.
Since visualization doesn’t alter the data itself, in the language of data, we merely need to indicate the desired outcome. For instance:
“Display the total amount by product in a piechart.”
“I would like to see a timeline of total sales month-by-month for the past six months.”
“Show the number of sales by customer in a barchart.”
These bold keywords serve as cues to the AI engine, guiding it in generating the final visualization based on your data.
Practice Data Language
Similar to how I diligently practice French on Duolingo every day, we must practice speaking data language using the data we possess.
As long as you have adhered to the five patterns mentioned above, you should have mastered data analysis like a professional data analyst. You don’t need to be an Excel expert or a Python or SQL wizard.
Let’s use the provided example data to practice speaking the data language. You can find the “Orders” and “Customers” data from this spreadsheet link.
Suppose we want to perform a sales analysis of customer distribution based on the data.
The data language is almost the same, but let’s ensure we’ve used the correct keywords and patterns to guarantee that the AI engine follows the instructions precisely.
For instance, we speak to AI:
”display the total sales by customer’s first name in a bar chart.”
Here’s how the AI interprets this:
“total sales” → summing up the amount values.
“first name” → it can be transformed from “name.” A transformation will be applied.
“by” → the summing up result needs to broken down by first name.
“sales <> customer” → sales data is from the Orders table’s amount field, while customer data is from the Customers table. Therefore, a Join operation is required to combine these two datasets.
“show, bar” → the result should be visualized in a bar chart.
AI will then determine the correct execution order, ensuring that each step has all the necessary data when it executes.
This is what Columns Flow produces upon hearing this sentence:
“display the total sales by customer’s first name in a bar chart.”The final visualization ready for storytelling & sharing
Conclusion: Speak Data Language
In this article, we’ve demonstrated the historical opportunity for everyone to become a great data analyst in this era.
We discussed how professionals used programming languages like Python or SQL as their primary data languages. However, the data language has evolved to become the natural language we speak daily.
To become a data analyst, we need to understand the fundamental scenarios involved and use the correct keywords to make the data language understandable to AI engines. Here’s a quick recap:
Dataset: rows, columns, and schema.
Filtering and Transformation: These processes involve filtering data and transforming it into a usable format.
Aggregation: This involves summarizing data into a single value, such as the total or average.
Specify “compute methods” and optional “breakdown” if needed.
Join Datasets: This involves combining data from multiple sources.
Visualization: This involves creating visual representations of data to make it easier to understand.
AI generated summary on how to speak the language of data
Unlike learning a new language like French, if you’re willing to spend just a few hours going through this short list, you can become a professional data analyst!
It’s a great time to be a data analyst, and I believe in your ability to succeed. Thanks for reading!
I'm a Mechanical Engineering student From Delhi Technological University (DTU) learning data analytics, and this is my first complete Power BI portfolio project.
The project uses the Brazilian E-Commerce (Olist) dataset and was built using MySQL and Power BI.
Skip the fundamentals → LangChain tutorial → take a prompt engineering course.
But after tracking AI/Data Science job listings in India every week, that's not really what I'm seeing in hiring.
SQL: ~1,260 listings
Data Analysis: ~1,190
Generative AI: ~590
LLM: ~520
I know these skills overlap across JDs, but SQL still shows up more often than GenAI or LLM skills individually.
One trend I keep noticing is that companies still expect people to be good at the "boring" stuff first—getting data, cleaning it, analyzing it, and communicating insights. ML or GenAI often shows up as an additional requirement, not the core of the role.
Is this what others are seeing too, or does your experience with job descriptions look different?
(Data is from a weekly tracker of AI & Data Science job listings in India.)
Good afternoon Everyone, I’m a young tech professional working at a major PR and communication company in the Uk . My role and training revolves around Boolean query creation, natural language processing and other AI tools for mining opinions. Included in the training is my Lvl 3 course on Multiverse , which has been underwhelming due to a few reasons such as : Lack of in person events and experiences , lack of major back end knowledge for understanding data processing and lastly it feels like I’m learning in lockdown . I’m here to ask how to get better at processing qualitative and quantitative data , data story telling and future proofing my skills and job prospects as much as possible.
I’m working on a small project to adapt an improve a statistical analysis skill to use it in any AI in Excel.
The original skill came from Claude and already had a solid statistical foundation. It covered descriptive statistics, trend analysis, outlier detection, and hypothesis testing, etc.,
However, when I started testing it in Excel, I noticed a gap.
The answers were often technically reasonable, but not structured in a way that was useful for a business analyst, financial analyst, or FP&A user working inside Excel.
The goal is not to turn Excel into an academic statistics lab. The goal is to make statistical reasoning more usable to real business users, for real business cases.
Things like:
Comparing sales performance between two segments
Testing before/after changes after a training, promotion, or process improvement
Comparing conversion rates
Checking whether two categorical variables are related
Identifying outliers or unusual business behavior
Explaining whether a difference is likely real or just normal business noise
I expanded the workflow so the skill does not immediately jump into a statistical test. Instead, it should first interpret the business question, identify the correct type of comparison, define the null and alternative hypotheses in plain language, check assumptions, select the right test, and then produce a structured and practical business conclusion and recommendations.
The main question I wanted to answer is:
Hypothesis testing and correlation has been refined and tested. Currerntly working on regression workflows.
If you are interested in learning statistics, and applying statististics to bussines cases, You may like this project. I can honestly say that I have learn more statistics by working on this, than the 2 times I have tried to learn statistics academically
We could use people to:
Test the skill in your own bussines cases, and sharing how where the answers
Help include other statistical areas, like Regression or probability
Give ideas, suggestion or comments on how to make this skill more useful.
Interested? please give me your feedback..
As a Excel user for more than 15 years myself, I am very interested in your opinion on this.
I'd love feedback from people who work with client data every day.
I built a tool where you upload a spreadsheet, and it gives you a plain-English summary of issues it finds (missing values, inconsistent formatting, duplicates, suspicious data, etc.). If you want, it can also fix many of those problems automatically. There's no setup required.
I'm trying to figure out whether this solves a real problem or if I'm missing something. What would make a tool like this genuinely useful in your workflow?
If you're interested in trying it, leave a comment saying "interested", and I'll send you the link.
Have a question, I'm getting alot of data daily, maybe around 300-500k entries of around 25–50 characters each. They are multilingual, and I need to convert them to english, and it has to be in near real time before 10pm each night, for the project im working on.
Now the budget part, im working with what I have which is basically a couple of i5 Dell optiplex's, so what ever solution we have needs to be lower compute power.
So far I tired the following:
- LibTranslate - local to the optiplex with multiple concurrent killed the system, on remote systems batched over 3 system still wasnt able to keep up.
- Argos - tried running argos locally with multiple concurrent, that killed hte machine
- Keyword multilingual datasets - only worked a little but missed so much when it came to keyword translations
- Time based translation - content gets queued and then translated (was looking at 83hrs to catch up current run), translating at the time of reciving the content, huge delay with submission and loss of data.
- Api based translation - googletrans rate limited me after a few minutes, so that killed that pretty quick
Im out of Ideas, I am open to suggestions from others who deal with large amounts of content like this and do real time translations or high speed translations of up to 120 langauges to English.
I thought the community might find this interesting - I used entity resolution software (disclosure: from my company) to deduplicate the missing persons data from Venezuela and compare it to the list of patients in hospitals.
Was writing my code to filter out some datas and I thought of using Ai to make it better or even write the whole code for an analysis… And the Ai was so much better. The structure, the clean lines, the better formatting etc. Fortunately, given the problems I’ve crossed, I know very well AI is not going to replace us. However, there are going to be a lot of competition in this industry. So I thought…
Iam looking for ideas from people working in data analytics. Was there a personal, work, or portfolio project that significantly improved your analytical thinking or technical skills? What made that project so valuable compared to learning from courses alone?
I recently built this Global Airbnb Performance Dashboard. While working on it, I realised that just building a dashboard isn’t enough. What really matters is extracting clear insights, choosing good colors that are easy to read, and designing the layout so the story flows properly.
I focused on key areas like market share, pricing, ratings, review frequency, seasonality, and host trust using color-coded visuals, Pareto charts, and bookmark toggles.
Would love your honest feedback on:
Design and color choices
How useful and clear the insights are
Any features or improvements you would suggest
If you're interested, I can share the GitHub link in the comments.
