Scaling Multimodal AI with Workload Specific AWS Compute – Perception Grid
Key Challenges
Perception Grid needed to process diverse digital assets, including documents, images, videos, and 3D models, while supporting both computationally intensive background processing and fast, interactive AI queries. The solution also needed to optimize compute costs and prevent redundant processing of previously embedded assets.
Key Results
Ankercloud implemented a scalable, workload-specific AWS architecture using Lambda for responsive AI queries and AWS Batch with EC2 for intensive asset processing. The solution enabled multimodal AI search, containerized processing, cost-optimized compute utilization, and reduced unnecessary processing through embedding tracking.
Overview
Perception Grid is creating a centralised platform for the management and searching of digital assets, including documents, photographs, 3D models, and other types of media. The platform required an AI-driven multimodal search feature which was to deal with these different kinds of assets and allow natural language queries to be used when searching their content. It had to be scalable and cost-effective while at the same time supporting both the computationally intensive background processing and the interactive AI needs.
To achieve these objectives, the solution was implemented using AWS managed compute services along two different workload routes. In the case of the interactive query path, Amazon API Gateway, AWS Lambda, Amazon Bedrock, and Amazon OpenSearch Serverless are used; while for the data-processing path, Amazon S3, AWS Batch, containerised Amazon EC2 computing, Amazon ECR, Amazon Bedrock, and OpenSearch Serverless are used.
Challenges
The client required handling a variety of digital assets, such as PDF files, movies, 2D pictures, and 3D models. The same compute pattern could not be used to process all of these assets. Specifically, before 3D models could be included in the multimodal retrieval method, they needed extra processing.
As a result, the solution required two different computations. While asset ingestion and embedding production required longer-running and more computationally demanding processing, user enquiries required a rapid, API-driven execution paradigm.
When choosing computing services, cost was another crucial factor. Because Amazon EC2 was thought to be more cost-effective for the workload, the team chose it over Amazon Fargate for these particular processing activities after evaluating the needs of the batch-processing workload during the design talks.
Additionally, the computational design had to prevent processing the same assets more than once. In order to guarantee that following processing solely concentrated on fresh data, a tracking system was implemented to determine which assets had already had embeddings created.
Solution
The architecture was designed with a clear separation between the interactive query pipeline and the data-processing pipeline.
Interactive Agent Compute
Through Amazon API Gateway, the query workflow exposes an API that calls an AWS Lambda function that hosts the agent workflow. The agent connects to Amazon OpenSearch Serverless to retrieve document and picture embeddings and employs Amazon Bedrock for model-based reasoning. Based on the user's request, the agent chooses which index to query.
Because the workload is request-driven, Lambda was suitable for this component. It gives the agent managed execution without requiring an application server to run continually. In order for the necessary application framework to function within the Lambda environment, the implementation also employs a Lambda-compatible wrapper around the application.
This enables the agent to call the managed retrieval and AI services needed to respond to the question while maintaining a lightweight user-facing compute path.
Batch Compute for Digital Asset Processing
When digital assets are uploaded to Amazon S3, the data-processing pipeline starts. With containerised workloads operating on Amazon EC2, the processing workflow orchestrates the processing jobs using AWS Batch. The processing environment uses container images that are stored in Amazon ECR.
The batch workload's characteristics led to the selection of the EC2-based processing environment. Instead of using the same computing service for both interactive and background workloads, this illustrated workload-based compute selection.
Job orchestration and the application containers themselves are kept apart by AWS Batch. This makes it possible for computationally demanding operations to run separately from the interactive agent workflow.
Containerized Processing
Amazon ECR is used to sustain and containerise the processing workloads. This enables the packaging of the dependencies needed for the multimodal processing workflow and offers a consistent deployment artefact for the batch-processing environment.
Managing 3D models was a major processing requirement. The implementation used PyRenderer and OpenGL to render each model into three 2D images that represented various viewing angles because the embedding technique was unable to directly consume the original 3D models. The visual embedding procedure could then include these produced representations.
Additionally, character descriptions were generated using mood-board JSON data using Claude 3.7. In order to save and index both textual and visual information for retrieval, the generated textual descriptions were mapped to the matching visual representations.
OpenSearch provides the retrieval layer for the multimodal RAG procedure, storing the created embeddings in distinct picture and textual indexes. Using the document index for script/text-based data and the picture index for visual and 3D-model data, the agent chooses the proper index based on the query.
Compute Selection and Optimization
The implementation shows that compute services are carefully chosen according to workload characteristics.
The interactive agent process used AWS Lambda because queries are event-driven and require managed execution, rather than dedicated servers. An API Gateway provides the interface into this computing layer.
The more complex data processing procedure used Amazon EC2 with AWS Batch. The team chose to use EC2 because it provided a less expensive computing option than Fargate for the batch-processing workload. This approach enabled the team to optimize compute economics without compromising the interactive query path.
Describe TCO Analysis Performed
The compute alternatives were analysed using the Perception Grid workloads' characteristics and expected execution pattern. For the intense batch-processing pipeline, the team examined the economics of container execution choices before choosing an EC2-based environment over Fargate for the processing workload. This method provided a more cost-effective compute model for longer-running and resource-intensive processing jobs, but Lambda remained suitable for event-driven agent query paths.
Instead of using a single compute model across the solution, the resulting architecture matched compute usage to workload needs. This helps to balance compute costs, processing requirements, and operational flexibility between interactive and batch workloads.
The container image repository for processing workloads is provided by Amazon ECR, which facilitates consistent deployment and the division of compute execution from application packaging.
As a result, the architecture does not use a single compute service for each workload. Containerised EC2-based batch processing handles computationally demanding asset preparation, while Lambda handles lightweight, request-driven processing.
To avoid re-processing current assets, the processing procedure also records the embeddings that have already been created. This minimizes unnecessary batch execution and the related consumption of compute when the asset repository is updated.
Business Outcome
The solution created a computational architecture that can separate interactive and background workloads and fulfil the multimodal AI requirements of Perception Grid.
The solution provides:
- Managed serverless compute for interactive agent requests through AWS Lambda.
- Batch-oriented compute for computationally intensive digital-asset processing through AWS Batch and EC2.
- Containerized workloads through Amazon ECR.
- Workload-specific compute selection based on execution requirements and cost considerations.
- Multimodal processing support for documents, images, and 3D models.
- Reduced redundant processing by tracking previously generated embeddings.
- Extensibility to support additional asset-processing and retrieval capabilities.
The final approach offers a workable compromise between dedicated container compute and managed serverless execution. While a computing architecture chosen for its cost-effectiveness isolates the heavier processing effort, the interaction approach stays responsive and lightweight.
Conclusion
The Perception Grid implementation shows how AWS managed compute services may be effectively applied to an agentic multimodal AI workload. The team chose AWS Lambda for API-driven agent execution and EC2-based AWS Batch processing for the heavier burden, purposefully separating interactive inference from computationally demanding data processing.
This strategy addressed the economic and operational features of Perception Grid's various AI workloads while giving it a scalable and adaptable compute basis.
