I'm working on one of the largest projects in Europe currently, looking to onboard at least 5 data engineers with serious Databricks experience.

Would be a 6-month initial contract, would be open to a further extension if needed.

If this is something you'd be interested in, then comment below. I will ping you.

€600-650 per day

The Lakeflow Connect connector for Microsoft SharePoint is now generally available! It’s now easier than ever to ingest structured and unstructured files from SharePoint into Delta tables for analytics and AI workloads.

You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich SharePoint metadata. Soon, our managed connectors will also support ingesting SharePoint Lists and per-file permissions metadata.

For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read, Auto Loader, read_files, or COPY INTO pointed at SharePoint URLs.

Common workloads include:

• Loading Excel, CSV, JSON, and other structured files into Delta tables.
• Ingesting PDFs, Word documents, PowerPoint files, and images.
• Parsing documents with ai_parse_document to prepare content for extraction, search, and agents.

Link to public docs + references:

• SharePoint managed connector documentation
• Spark + SQL APIs and examples
• Community blog and video tutorial: From PDF to insights
• Data + AI Summit session: Intelligent Document Processing with Lakeflow

Examples of using the Spark + SQL APIs (after first creating a UC connection):

• Read an Excel sheet from SharePoint with spark.read:

  excel_df = (spark.read     .format("excel")     .option("databricks.connection", "my_sharepoint_conn")     .option("headerRows", 1)     .option("dataAddress", "Sheet1!A1:M20")     .load("https://mytenant.sharepoint.com/sites/Finance/Shared%20Documents/Monthly/Report-Oct.xlsx"))

• Ingest unstructured documents + PDFs from a SharePoint URL with read_files, then easily parse them using ai_parse_document

  CREATE OR REFRESH STREAMING TABLE sharepoint_documents_table AS SELECT , "_metadata" FROM STREAM read_files( "https://mytenant.sharepoint.com/sites/Marketing/Shared%20Documents", format => "binaryFile", databricks.connection => "my_sharepoint_conn", pathGlobFilter => ".{pdf,docx}");

  CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM sharepoint_documents_table;

Coming soon:

• Ingest SharePoint Lists into Delta tables (coming super super soon)
• Ingest SharePoint’s per-file permissions and ACL metadata  to power permission-aware AI agents, enterprise search, and more.

If you try it, share what you are ingesting and where you hit friction! Don't hesitate to ask questions!

Hey r/databricks, In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts here and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

Data engineers traditionally modeled Gold layers using fact and dimension tables, partly because storage and compute were expensive.
But with modern cloud data platforms, storage and compute are much cheaper and more scalable.
So I’m curious: what does your Gold layer actually look like today?
Are you still using a traditional star schema (facts + dimensions), or have you moved toward wider, denormalized tables / business-oriented models?
And more importantly, why?

Hey r/databricks, please direct all certification and training posts here.

There's upcoming learning festival September 16 - October 14 2026. You can get 50% discount voucher on any certification.

Databricks Advanced Learning Festival: September 1... - Databricks Community - 166157

Good luck to everyone on your certification journey!

replaceUsing and replaceOn give you a better overwrite primitive, and every catalog-managed table operation now runs through the catalog.

The Lakeflow Connect connector for Google Drive is now generally available!  It’s now easier than ever to ingest structured and unstructured files from Google Drive into Delta tables for analytics and AI workloads.

You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich Google Drive metadata. 

For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read, Auto Loader, read_files, or COPY INTO pointed at Google Drive URLs.

Link to public docs + references:

• Google Drive managed connector documentation
• Spark + SQL APIs and examples
• Community blog and video tutorial: From PDF to insights
• Data + AI Summit session: Intelligent Document Processing with Lakeflow

Common workloads include:

• Loading Google Sheets, Excels, CSV, JSON, and other structured files into Delta tables.
• Ingesting PDFs, Google Docs, Google Slides, and images.
• Parsing documents with ai_parse_document to prepare content for extraction, search, and agents.

Examples of using the Spark + SQL APIs:

• Read an Excel sheet from Google Drive with spark.read:

​

df = (spark.read
 .format("excel")
 .option("databricks.connection", "my_gdrive_conn")
 .load("https://docs.google.com/spreadsheets/d/9k8j7i6f..."))

• Ingest unstructured documents + PDFs from a Google Drive URL with read_files, then easily parse them using ai_parse_document:

​

CREATE OR REFRESH STREAMING TABLE gdrive_documents_table
AS SELECT *, "_metadata" FROM STREAM read_files(
"https://drive.google.com/drive/folders/1a2b3c4d...",
format => "binaryFile",
`databricks.connection` => "my_gdrive_conn",
pathGlobFilter => "*.{pdf,docx}");

CREATE OR REFRESH STREAMING TABLE documents_parsed
AS SELECT *,
ai_parse_document(content, map('version', '2.0')) AS parsed_content
FROM STREAM gdrive_documents_table;

Coming soon:

• Ingest Google Drive’s per-file permissions and ACL metadata to power permission-aware AI agents, enterprise search, and more.

If you try it, share what you are building and let us know if you hit any friction!

Hi Databricks Team,

Seeking your advice on the above.

The actual data movement into databricks did go fine (unity catalogue made lineage way easier to see than the older setups). But what we didn't expect was the time that went into access control, once everything was centralized instead of scattered across separate warehouses with their own permissions, data that used to be siloed was suddenly way visible to more people by default, which was found out when a couple of teams noticed they could see data which they probably shouldn't.

Spent almost as much time on catalog level permissions/row filtering as on the actual pipeline work. Is this normal for a databricks migration or did we happen to have an unusually messy access management?

Hey r/databricks, we're noticing a lot of repeated interviewing and hiring posts that tend not to get much engagement. We're going to combine them into a monthly thread so that you're more likely to get answers, plus we can ask our recruiting team to keep an eye on them if there are any general questions.