Name: What ran on Databricks, and what it cost me
Author: recued-core

What ran on Databricks, and what it cost me

by recued-core v2 36 views

Use “What ran on Databricks, and what it cost me” in Recued. Break down recent Databricks query history by its actual status and show the warehouse fleet beside it. It reads information without changing data in the connected service.

How it works 39 steps

Inspect the data fetches, transforms, gates, and output this recipe runs.

Process (39 steps)

■ window clamp

Clamp setting max results between 1 and 1000

◆ history_raw recued-core.databricks.query_history.list

List query history on databricks

■ runs default

Apply default

■ have_runs template

Generate text from a template

fail: step.runs is_empty

■ total count

Count items in runs

■ finished filter

Filter by condition

■ finished_count count

Count items in finished

■ failed filter

Filter by condition

■ failed_count count

Count items in failed

■ canceled filter

Filter by condition

■ canceled_count count

Count items in canceled

■ in_flight filter

Filter by condition

■ in_flight_count count

Count items in in flight

■ success_pct_raw math

Calculate: {{step.finished_count}} / {{step.total}} * 100

■ success_pct_round round

Round success pct raw to 0 decimal places

■ with_submitter filter

Filter by condition

■ submitter_names pluck

Extract user_name from each item

■ submitters unique

Remove duplicates from

■ submitter_count count

Count items in submitters

■ with_executor filter

Filter by condition

■ executor_names pluck

Extract executed_as_user_name from each item

■ executors unique

Remove duplicates from

■ executor_count count

Count items in executors

■ by_duration sort

Sort by duration

■ slowest_five slice

Take a subset of

■ slowest max_by

Find the maximum by duration

■ slowest_ms default

Apply default

■ slowest_s_raw math

Calculate: {{step.slowest_ms}} / 1000

■ slowest_s round

Round slowest s raw to 1 decimal places

◆ wh_raw recued-core.databricks.warehouse.list

List warehouse on databricks

■ warehouses default

Apply default

■ wh_count count

Count items in warehouses

■ wh_running filter

Filter by condition

■ wh_running_count count

Count items in wh running

■ wh_never_stop filter

Filter by condition

■ wh_never_stop_count count

Count items in wh never stop

■ card to_summary

Format results as a summary card

■ failed_table to_table

Format results as a data table

■ slow_table to_table

Format results as a data table

▼

Output

summary card

table failed_table

table slow_table

Settings 1 configurable

Configurable at install. Defaults shown — change them anytime in Recued.

max results setting = 100

Uses (2 operations)

Pack operation List query history on databricks recued-core.databricks.query_history.list

Pack operation List warehouse on databricks recued-core.databricks.warehouse.list

Trust & control

What installing this recipe would let it do. Recued grants these permissions at install — review them there before approving.

Permissions it requires

Read your Databricks connection

Declared by the recipe — Recued grants these at install, where you review them before approving.

About

recued-core

Details

39 steps 1 configurable settings recipe_id: review-query-history-databricks

What ran on Databricks, and what it cost me

Uses (2 operations)

Trust & control

Permissions it requires

About

Tags

Details