Amazon Web Services now lets SageMaker real-time inference endpoints split a single Nvidia GPU into as many as seven hardware-isolated partitions using Nvidia's Multi-Instance GPU technology, known as MIG. AWS announced the feature in a November 6 blog post, pitching it as a way to run several smaller models on one accelerator instead of leaving most of a large GPU idle.
Partitioning is supported on four instance families. P4d instances with Nvidia A100 40GB GPUs and P5 instances with H100 80GB GPUs can each be carved into up to seven slices. G7e instances, built on the 96GB RTX PRO 6000 Blackwell Server Edition, support up to four, and G7 instances with the 32GB RTX PRO 4500 Blackwell support two. The feature is available in US East (N. Virginia), US East (Ohio), US West (Oregon), Singapore, Tokyo, Frankfurt and Ireland.
The capability works only on endpoints built with SageMaker inference components. AWS said single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned. There are no new API operations or IAM actions: customers add an AcceleratorPartitionLayouts field to the production variant in CreateEndpointConfig, then request a specific profile, such as mig-4g.20gb, in each inference component's compute requirements. Endpoints that omit the new fields behave as before.
Key facts
- 01SageMaker real-time endpoints can now split one Nvidia GPU into up to seven isolated MIG partitions.
- 02Supported on P4d (A100) and P5 (H100) instances, plus G7e (up to four slices) and G7 (two), in seven AWS regions.
- 03An H100 mig-3g.40gb slice was 18% to 34% cheaper per million output tokens than a whole A100 above a crossover point.
- 04Customers still pay the full instance price; carving added roughly 17 seconds to provisioning in AWS tests.
- 05Partitioning works only with inference components, not single-model or serverless endpoints.
Layouts are set when an endpoint is created and apply uniformly to every GPU on the instance. AWS gave the example of an ml.p4d.24xlarge configured with mig-4g.20gb, mig-2g.10gb and mig-1g.5gb profiles, which turns the instance's eight A100s into 24 partitions. Changing the carve later requires a new endpoint and a traffic migration, a component cannot fall back to a whole GPU if its requested profile is missing, and no partition can span more than one accelerator.
Pricing does not change. SageMaker customers still pay the full hourly rate for the GPU instance, so any savings come from packing more work onto it. In AWS testing, carving the GPUs added roughly 17 seconds to endpoint provisioning.
To show the tradeoff, AWS benchmarked one H100 mig-3g.40gb partition against a whole A100 40GB, each serving Llama-3.1-8B-Instruct across five traffic shapes at concurrency levels from 2 to 512. The H100 partition produced more output tokens per second in every scenario, by 53% to 89% at the load where each scenario's cost advantage was largest. Inter-token latency was 8.9% to 42.1% lower, and time to first token was 5.6% to 47.5% lower.
On cost per million output tokens, the partition came out 18% to 34% cheaper once traffic passed a crossover point, which AWS put at a concurrency of 64 for 100-token prompts and 4 for 8,000-token prompts. Below that line the whole A100 stayed cheaper, and AWS framed the results as a case-by-case comparison rather than a blanket recommendation. At the top of the concurrency ladder on 6,000- and 8,000-token workloads, the A100 failed between a third and half of requests against a serving deadline, while the partition failed none.
Salesforce worked with AWS on the feature. According to the blog post, the company helped define requirements and validated the initial solution, and it is now preparing to test partitioning against a representative production workload.
“MIG would allow us to consolidate workloads safely, significantly improve GPU utilization”— Richard Chang, Software Engineering Architect, Salesforce
AWS based its cost math on prices of $25.25 per hour for ml.p4d.24xlarge and $63.30 per hour for ml.p5.48xlarge in US West (Oregon). It also flagged one catch for P4d users: the mig-1g.10gb profile needs a newer Nvidia driver than the default image ships, so customers must set the al2023-ami-sagemaker-inference-gpu-4-1 inference AMI version or the carve fails without an error naming the profile.
AWS published a GPU partitioning guide in the SageMaker documentation and a sample notebook in its aws-samples GitHub repository for teams that want to test the new layouts.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
