Qwerky: Model distillation and training in HPC-based environment
"Tech 42 was a strong and trusted partner throughout our model distillation POC. Despite external challenges around AWS GPU capacity, their team worked closely with us to adapt the approach, pivot the infrastructure strategy, and keep progress moving forward. As part of this, we transitioned the distillation workloads to AWS using an HPC-based setup, enabling distributed training at scale. Once capacity constraints were addressed, we were able to accelerate experimentation and successfully validate distributed training.
The team demonstrated deep expertise in large-scale model training, AWS infrastructure, HPC environments, and experimentation workflows. They collaborated closely with our engineers, communicated clearly through Slack and weekly syncs, and were thoughtful about incorporating feedback as the project evolved. Importantly, they helped us validate that our distillation approach was feasible at scale and laid the groundwork for a future production-ready HPC environment on AWS.
We appreciated Tech 42’s flexibility, transparency, and technical guidance, and we look forward to continuing the partnership as we move into the next phase."
Evan Owen
Co-founder & CTO
Project summary
QWERKY AI needed to optimize conversational AI models for performance and efficiency. Tech 42 implemented model distillation using Llama 3.2 8B in AWS SageMaker, starting with a proof-of-concept and evolving to production-scale infrastructure. The team configured the SageMaker environment, established training and validation datasets, and deployed the distilled model. When AWS GPU capacity became constrained, Tech 42 adapted the approach by transitioning to an HPC-based distributed training setup on AWS. This enabled experimentation to accelerate and validated that the distillation approach would work at scale. Deliverables included a functional distilled model accessible via boto3 SDK, complete documentation and training code in GitHub, performance assessment reports, and knowledge transfer sessions. The work laid the groundwork for QWERKY's production-ready HPC environment on AWS.
