Kubernetes v1.36 has promoted a feature that lets you modify container resource requests and limits in the pod template of a suspended Job to beta. First introduced as alpha in v1.35, this capability gives queue controllers and cluster administrators the power to adjust CPU, memory, GPU, and extended resources on a Job while it’s paused—before it starts or resumes running. Whether you’re managing batch processing or machine learning workloads, this update eliminates the need to delete and recreate Jobs when resource requirements change after submission. In this article, we break down the eight most important aspects of this new beta feature.

1. What Is Mutable Pod Resources for Suspended Jobs?
2. Why This Feature Matters for Batch and ML Workloads
3. The Problem It Solves: Immutability No More
4. How Queue Controllers Benefit
5. Real-World Example: Adjusting GPU Resources
6. Step-by-Step: How to Use It
7. How It Works Under the Hood
8. What’s Next and Best Practices

1. What Is Mutable Pod Resources for Suspended Jobs?

In Kubernetes v1.36, the ability to modify container resource requests and limits inside the pod template of a suspended Job has graduated to beta. Previously, once a Job was created, its resource specifications were frozen—changing them required deleting the Job and starting fresh. Now, while a Job is in a suspended state (spec.suspend: true), you can update fields like resources.requests.cpu or resources.limits.memory directly. The change is transparent: the Job remains suspended until the modifications are complete, then you can resume it with the new values. This feature is a game-changer for dynamic resource management.

2. Why This Feature Matters for Batch and ML Workloads

Batch processing and machine learning jobs often have resource needs that aren’t known precisely at creation time. The optimal allocation depends on real-time cluster capacity, queue priorities, and availability of specialized hardware like GPUs. Without mutable pod resources, queue controllers had to guess or conservatively request resources, leading to inefficiencies. Now, administrators can fine-tune resource requests after a Job is submitted but before it starts—matching workload demands to actual cluster conditions. This reduces waste and improves throughput, especially for large-scale training jobs that could use extra memory or fewer GPUs based on current load.

3. The Problem It Solves: Immutability No More

Before Kubernetes v1.36, resource requests and limits in a Job’s pod template were immutable. If a queue controller like Kueue determined that a suspended Job should run with different resources, the only recourse was to delete and recreate the Job. This caused loss of associated metadata, status, or history. Worse, CronJob instances that encountered heavy cluster load would fail outright, unable to run with reduced resources. The new feature solves this by allowing incremental adjustments—keeping the Job’s identity intact while adapting to changing environments. It’s a simple but powerful change that eliminates painful workarounds.

4. How Queue Controllers Benefit

Queue controllers—such as Kueue, Volcano, or custom schedulers—can now dynamically optimize resource allocation. When a Job is suspended, the controller can inspect available cluster resources and update the Job’s pod template accordingly. For example, if only 2 GPUs are free instead of the requested 4, the controller can reduce the GPU request before resuming. This keeps Jobs from being stuck in pending states or failing due to resource shortages. The controller also retains full visibility into the Job’s history and labels, ensuring auditability. The beta status means this capability is stable enough for production use cases.

5. Real-World Example: Adjusting GPU Resources

Consider a machine learning training Job that initially requests 4 GPUs. The YAML might look like this (simplified): resources.requests['example-hardware-vendor.com/gpu']: '4'. With the new feature, a queue controller can update that to 2 GPUs if cluster capacity demands it. The Job remains suspended (spec.suspend: true) while the change is made. Once updated, the controller sets spec.suspend: false and the Job creates pods with the adjusted resource specs. The original Job object—including its name, annotations, and status—survives intact. This scenario highlights the feature’s value for elastic, preemptible workloads.

6. Step-by-Step: How to Use It

Using mutable pod resources is straightforward. First, ensure your cluster runs Kubernetes v1.36 with the MutableResourcesForSuspendedJobs feature gate enabled (default on). Next, create a Job with spec.suspend: true and define your initial resource requests. Later, use kubectl edit job or a controller to change the spec.template.spec.containers[].resources fields. The API server validates the changes only against the pod template schema—no new resource types are needed. Finally, resume the Job by setting spec.suspend: false. The pods will be created with the updated resources. Remember: modifications are only allowed while the Job is suspended.

7. How It Works Under the Hood

The Kubernetes API server relaxes the immutability constraint on pod template resource fields specifically for suspended Jobs. No new API types or custom resources were introduced; the existing Job and PodTemplateSpec structures accommodate the change through conditional validation. When a Job is not suspended, the resource fields remain immutable, preserving safety. The beta promotion means the feature is deemed stable enough for production, with thorough testing and documentation. This design keeps the Kubernetes API clean while enabling critical flexibility for dynamic workload management.

8. What’s Next and Best Practices

As this feature matures, expect broader adoption in queue controllers and CI/CD pipelines. Best practices include using it for “slow start” scenarios—letting CronJob instances run with reduced resources instead of failing. Also consider combining it with resource quotas and priority classes to prioritize critical Jobs. Monitor Job status changes to ensure updates are applied before resumption. Finally, test in non-production environments first, as unexpected resource scaling could impact co-located workloads. The Kubernetes community continues to refine this feature, so stay updated with future releases for any enhancements.

Conclusion

Mutable pod resources for suspended Jobs is a practical, long-awaited enhancement that brings flexibility to Kubernetes job scheduling. By enabling resource modifications without Job recreation, it streamlines batch and ML workflows, reduces operational overhead, and improves cluster utilization. As a beta feature in v1.36, it’s ready for real-world use—just remember to keep Jobs suspended while making changes. Whether you’re a cluster administrator or a developer, this update empowers you to adapt to changing conditions without losing Job history or metadata. Try it out and see how it transforms your resource management.