SkyPilot and Hugging Face launched storage with no egress fees

Hugging Face and SkyPilot on July 7, 2026 launched cloud storage integration: developers can now run AI workloads on any of 20+ cloud services, reading model weights and datasets directly from Hugging Face without egress charges.

Why did egress traffic become a problem

Teams training models often maintain GPUs with multiple cloud providers simultaneously—wherever free resources are found. Storing data in each cloud is expensive and inconvenient, and transferring data between providers costs about $0.09/GB at AWS. With datasets in hundreds of gigabytes, egress charges quickly exceed computation costs.

Key facts about the integration:

• Announcement – July 7, 2026, joint development by SkyPilot and Hugging Face teams
• Hugging Face Storage: $12–18/TB/month, egress traffic—free
• AWS S3: ~$23/TB plus $0.09/GB for egress traffic
• Model loading speed when mounted: up to 500 MB/s, cold start ~30 seconds
• One-line installation: `pip install "skypilot[huggingface]"`

How mounting via hf:// works

New `store: hf` backend in SkyPilot configuration accepts addresses in `hf://` format—repositories with models, datasets, or buckets. Authorization passes through `HF_TOKEN` environment variable.

Two access modes are supported. MOUNT—streaming read via FUSE: only requested bytes are loaded, without copying the entire file. COPY—pre-loading via `huggingface_hub` with disk cache. Hugging Face buckets are open for read and write (for checkpoints), model and dataset repositories—read-only with optional revision pinning.

Hugging Face also added FUSE patches allowing mounting without superuser rights—a standard Kubernetes requirement where root is unavailable.

How fast are checkpoints written

Teams tested fine-tuning of Qwen3.5-4B model on Multilingual-Thinking dataset across three clouds simultaneously: AWS with L40S GPU showed ~168 MB/s, GCP with L4—~123 MB/s, Lambda with H100—~112 MB/s. Variance between providers is minor and fits within typical workload ranges.

Additional gains provided by deduplication via Xet technology: files are split into ~64 KB blocks by content, and repeated loading skips already-known blocks. In practice, adding 10,000 rows to a Parquet table of 100,000 rows transferred ~10 MB instead of ~106 MB, and re-loading an 8.43 GB blob took 8 seconds instead of 24.

What this means

Egress charges were a hidden barrier forcing teams to keep data and compute in one cloud. SkyPilot and Hugging Face Storage integration removes this barrier: teams can choose GPUs by availability and price from any provider without losing money to traffic.

Frequently asked questions

Can you write checkpoints back to Hugging Face?

Yes—Hugging Face Buckets support both read and write. Model and dataset repositories are read-only. When writing from cloud to Hugging Face, standard cloud provider egress charges apply—only ingress from Hugging Face is free.

What infrastructures does SkyPilot support?

SkyPilot works with 20+ cloud providers, Kubernetes clusters, Slurm, and on-premises infrastructure. Base image must have glibc 2.34+ and `/dev/fuse` access.

How many clouds does

Hugging Face and SkyPilot integration support? The integration supports 20+ cloud providers for running AI workloads.

Why is egress traffic expensive when training models?

Data transfer between cloud providers costs approximately $0.09 per gigabyte at AWS. With datasets sized in hundreds of gigabytes, egress charges quickly grow.