Automate CSV to Parquet with Hugging Face Buckets, Webhooks & Jobs

Files keep landing in your Hugging Face Storage Bucket. Here is how to process each one automatically, without downloading anything. In this live-coding video we set up a webhook on a bucket that triggers a Hugging Face Job whenever a new file arrives. The Job runs the `optimize-parquet.py` uv script, converts the new CSV to optimized Parquet, and writes it to a second bucket. We upload a 10,000-row slice of the NYC taxi dataset, and the triggered Job finishes in about 8 seconds. For data engineers already using Buckets who want to automate what happens next. Topics covered: - Hugging Face Storage Buckets with the hf CLI - Running a uv script as a Hugging Face Job - Webhooks that trigger Jobs on bucket changes - Converting CSV to Parquet automatically - Checking Job status, logs and output Resources: - Webhooks guide (bucket to Jobs): https://huggingface.co/docs/hub/webhooks-guide-bucket-jobs - optimize-parquet.py script: https://huggingface.co/datasets/uv-scripts/data-processing/raw/main/optimize-parquet.py - My notes and demo code: https://github.com/prpatel/simple-buckets-demo - Hands-on with Buckets (article): https://huggingface.co/blog/prpatel/hands-on-with-buckets Chapters: 0:00 Intro 0:16 Why automate bucket processing 0:39 Input and output buckets 1:39 Define the Job 2:20 Create a webhook on the bucket 3:21 Upload a new CSV 4:01 Check the Job and its output 5:04 Wrap up