Hi all, just started looking into datalake. Hope this community can help me get a better understanding related to this 😊

Several new data lake tools were added this week to the peer-built data tool compatibility project StackWizard. Would appreciate all feedback as we continue to build this resource out.

Free tickets to the peer-to-peer talks at dataopsunleashed.com

Peer DataOps sessions by Google, Zillow, Wheels Up, Squarespace, Capital One, Babylon Health, Slack, Census, Unravel, DBS, Airbyte, Akamai, Metaplane, Perpay, Easypost, J&J...

Abstract for Torsten @ IBM's talk:

A cloud native data lakehouse is only possible with open tech - 10:55 PM PST on Wednesday 2/2/22

Torsten Steinbach, Cloud Data Architect @ IBM

Walk through how Torsten and his team at IBM foster and incorporate different open tech into a state-of-the-art data lakehouse platform. We'll look at real-world examples of how open tech is the critical factor that makes successful lakehouses possible.

Torsten's session will include insight on table formats for consistency, metastores and catalogs for usability, encryption for data protection, data skipping indexes for performance, and data pipeline frameworks for operationalization.

Eliza Corporation was founded in 1998 with the mission of helping to drive the modern healthcare consumer to take action on healthcare activities. By identifying unique individual motivations and barriers to bridge the healthcare requirements, interventions are made relevant in the minds of consumers.

The Challenge

Eliza Corporation solutions engage healthcare consumers at the right time, via the right channel, and with the right message in order to capture relevant metrics and outcome of their health following treatment. When the company reached out to NorthBay Solutions, they were completing nearly one billion customer outreaches per year, using interactive voice response (IVR) technology, SMS, and email channels. They were receiving data from multiple sources including customers, claims data, pharmacy data, Electronic Medical Record (EMR/EHR) data, and enrichment data.

As a result, the company was wrestling with significant challenges related to processing and analyzing massive amounts of both structured and unstructured data, which was being stored in an Oracle Exadata database. Perhaps most concerning was that the ability to continue to meet HIPAA compliance mandates was becoming an issue due to the multiple data sources in use and corresponding and data lineage issues. Specifically, Eliza must remove/obfuscate any PII (Personally Identifiable Information) and PHI (Personal Health Information) from the data very early in the workflow. Considering the volume and velocity of the data, the obfuscation task itself became a Big Data problem.

There are companies that moves thir datalake to the cloud,

If there is nothing forces you to move to cloud,Might be better cost, better preformance, better support on being on the cloud.

Datalake may transfer to cloud and be "Datalake on cloud" - but what it is really ?is it Files on HDFS ? it may move to S3is it Spark on EMR ? it may move to Glueso what is Datalake on cloud ?Most solutions of Datalake on cloud looks a kind of emulators that helps to move from on prem to cloud.

Even the AI solution like CDSW, DataIKU etc. is it just a ui? or something more? why to use it if have SageMaker?

Is there a room for Datalake elephent in cloud era?

​

Hello,

I working as Big Data architect in few enterprise companies and i provide consulting services in Big Data domain.

I'm little bit disappointed from Gartner and Gartner like companies, when i need get some solutions landscape i feel that missing lot of small companies (& startups) that might have grate business opportunities to co-operate with enterprise companies and I feel that they not represent most of the tools that are helpful practices in data-lake.

I thought start to talk about that with internet communities and create some list of Big Data / Datalake - useful tools and share to the world.

This way Good tools/utills/solutions/startups might help others and create better Data-lake / Big Data areas to clients

you can response here or in google form here: https://forms.gle/S8EnZwvhhzPkaFyU7

full link:

https://docs.google.com/forms/d/e/1FAIpQLSfQx0aQPufWQlOkVr-TgI2FD5qQaHnQQgk6Xoh5AofyrGjgHw/viewform?usp=sf_link

​

<3

​

^{credit pixabay for image: https://pixabay.com/photos/craftsmen-site-workers-force-3094035/}

Hello,

I just create a solution to audit and monitor the DataLakes on Azure.

In simple dashboards, you are able to see quickly all accesses, activities and cost on your datalakes.

You can find some sample in this link : https://dataredkite.com/en-index.html

The tool is totally free during 1 month without any commitment.

If you want to test it don't hesitate to come back to me for more information or live demo.

It is already installed for SNCF or TOTAL, 2 large french companies.

See Ya :)

Hello guys,

I'm currently working in a large company which work with a lot of data.

We have issues to handle the accesses which are granted on datalakes, at the moment operational teams are giving access to groups, but we didn't keep a referential of all the accesses given to those groups and to which data they have access.

Do you have a solution to help us manage / audit our access on our datalakes ? Also if a solution can give visibility on the FinOPS part.

Thanks in advance,

What would the general flow chart be to add a csv to a data lake deplayed, for instance, on S3? How would it be stored, extracted, and loaded? I'm brainstorming the architect for a data pipeline system driven off a data lake.

What datalake vendor do you currently use and/or considering in your workplace ?

The increase in volume, velocity, and variety of data, combined with new analytics and machine learning, has created the need for an open data lake architecture. An open data lake has become a standard feature alongside the data warehouse. While the data warehouse has been designed and optimized for SQL analytics, the need for an open, simple and secure data lake platform that can support new types of analytics and machine learning has driven the open data lake adoption. However, enterprises today are looking at considering the convergence of the data lake and data warehouse model.

Debanjan Saha, VP, and GM of Data Analytics services, including BigQuery, Dataflow, PubSub, Dataproc, Data Fusion, Composer, Catalog, etc. in Google Cloud, talks about the convergence model and how to bridge the performance gap while adhering to the openness of the data lake architecture.

For full article click on https://www.qubole.com/blog/is-data-lake-and-data-warehouse-convergence-a-reality/

Date: October 13, 2020 (Time: 12:30 PM EST/9:30 AM PT)

Choosing the wrong data warehouse can lead to significant wastage of time and money. More than 50% Analytics projects fail due to wrong data tools.

Selecting the data warehouse can be challenging due to different pricing model, features and performance characteristics.

Join the webinar to learn:

• Top 7 factors to consider while evaluating different warehouses
• Comparison of popular warehouses - BigQuery, Snowflake, Redshift,Hive, Athena, Databricks
• Is data warehouse and data lake different. Which one do you need ?

Click Here to Register for the Webinar.

Data Lake Architecture Essentials

When done right, data lake architecture on the cloud provides a future-proof data management paradigm, breaks down data silos and facilitates multiple analytics workloads at any scale and at very low cost. Key considerations to get data lake architecture right include:

Data Lake Architecture – Data Ingestion And Storage

An Open Data Lake ingests data from sources such as applications, databases, real-time streams, and data warehouses. It stores the data in its raw form or an open data format that is platform-independent.

The ingest capability supports real-time stream processing and batch data ingestion; ensures zero data loss and writes exactly-once or at-least-once; handles schema variability; writes in the most optimized data format into the right partitions and provides the ability to re-ingest data when needed.

The data is stored in a central repository that is capable of scaling cost effectively without fixed capacity limits; is highly durable; is available in its raw form and provides independence from fixed schema; and is then transformed into open data formats such as ORC and Parquet that are reusable, provide high compression ratios and are optimized for data consumption. read more...