AI DevOps & Infrastructure
Model Evaluation
AIQ Markets: Open source LLM serving and benchmarking infrastructure
"AIQ Markets operates an AI-powered intelligence platform for U.S. corporate bond markets, and as we looked to scale, we needed a way to evaluate open source language models against our existing model dependencies — on infrastructure we could control. The challenge was that our internal benchmarking framework could only integrate API-accessible models. We had no way to self-host and benchmark open source alternatives, which limited our ability to make confident, data-driven decisions about our AI model strategy.
Tech 42 delivered a flexible, repeatable model serving infrastructure on AWS purpose-built for this exact need. The solution included a HuggingFace model download pipeline triggered via S3, FSx for Lustre for high-performance model weight storage, vLLM serving containers running on GPU EC2 instances via ECS, an OpenAI-compatible API endpoint that plugged directly into our existing benchmarking framework, and a full CI/CD pipeline via GitHub Actions for Terraform-driven infrastructure updates. The result was a self-hosted benchmarking capability that gave us two levers we didn't have before — full control over which model to evaluate and full control over the infrastructure it runs on.
The impact has already been meaningful, even ahead of production deployment. We now have the ability to rigorously benchmark open source models against our existing dependencies on our own terms, rather than being limited to whatever was API-accessible. While we haven't yet moved this into production — largely due to the RAM costs of running some open source models at scale — we have the infrastructure and optionality in place to move the moment the economics make sense. That kind of flexibility is exactly what we needed for a platform built on real-time financial decision-making.
Beyond the technical delivery, the engagement itself was seamless. Communication was efficient, scheduling was easy, and the team was consistently on point. Tech 42 exceeded our expectations from start to finish."
Kevin Rutter
CEO
Project summary
AIQ Markets operates an AI-powered intelligence platform for U.S. corporate bond markets. Its benchmarking framework could only evaluate models available over an API, so open source models could not be tested. Tech 42 built self-hosted model serving and benchmarking infrastructure on AWS, defined in Terraform. A HuggingFace model download pipeline runs on AWS Batch and is triggered by configuration file uploads to Amazon S3. Model weights are stored on FSx for Lustre and synchronized back to S3. vLLM serving containers run on GPU EC2 instances through Amazon ECS, with tensor parallelism and an Application Load Balancer exposing an OpenAI-compatible API endpoint. API keys are held in AWS Secrets Manager. GitHub Actions applies targeted Terraform updates. The benchmarking stack is ephemeral and can be destroyed and redeployed without affecting core resources. AIQ Markets can now select which open source models to evaluate and control the infrastructure they run on.
