Things You Wish You Had Known Earlier About Databricks Performance

A big thank you to Jixin Jia (Gin) for a brilliant presentation at the August Sydney Databricks meetup. It was full of practical tips to improve speed with a set of Databricks Notebooks to back it up as proof! (download them below if you want to try them out).

I highly recommend checking out his blog, the Book of Architecture – you are guaranteed to learn something new!

Here’s what was covered:

The sweet spot between cluster performance and cost
More about partitioning
How to manage small files (A common problem that we face regularly with Data Lakes)
The use of Delta Lake
Learn things about Databricks/Spark Caching that you didn’t know existed

Download the Slides & Databricks Notebook

Fill in the form below to download the goodies…

Rodney Joyce

Azure-certified Data Architect with a focus on delivering business value and guiding customers through the maze of analytical architectures, design and implementation activities.

Experienced in setting up modern data platforms with advanced predictive analytic workloads. Brings strong people skills and a devops-centric, entrepreneurial approach to Enterprise software delivery.