Databricks performance lessons from a Sydney practitioner session

In August 2020, Jixin Jia presented a practical Databricks performance session to the Sydney Databricks meetup. This page preserves the topics and references from that event.

It is not a current product guide. Databricks runtimes, features, and recommendations have changed since the session, so test every idea on the runtime and workload you operate today.

What the session covered

The talk connected performance to cost rather than treating speed as the only goal. Its main topics were:

• finding a useful balance between cluster resources and job duration
• choosing partitioning that matches data volume and query patterns
• managing the small-file problem in a data lake
• using Delta Lake for reliable table operations
• understanding when Spark caching helps or adds pressure

Those themes remain useful because each one affects how much data Spark reads, moves, or keeps in memory.

Turn each tip into a benchmark

Use a representative query and dataset before changing configuration. Capture runtime, input size, shuffle, spill, task distribution, and compute cost.

Change one factor at a time. A larger cluster may finish sooner but cost more. More partitions can improve parallel work until scheduling and small-file overhead dominate. Caching can help repeated reads but compete with execution memory.

For a focused example, read our small-file and Delta Lake guide. Treat its 2020 settings as historical and validate them against current Databricks guidance.

Original references

Jixin’s Book of Architecture and LinkedIn profile were linked in the original recap. The Sydney Databricks User Group listing is retained as an event reference.

The imported page contained an inactive form for slides and notebooks. It has been removed because no working resource link was present. This article does not claim that those files are still available.