
Tech
10 min read

Building an AI model is one challenge. Making it fast enough, cheap enough and accurate enough to run in a real production environment is an entirely different one. A model that performs brilliantly in a research environment can grind to a halt under real-world load, slow inference, bloated memory usage, sky-high compute costs. That's the problem AI model optimisation solves. And for businesses deploying AI at scale, getting this right isn't optional.
Regardless of whether it is an in-house language model, recommendation algorithm or computer vision process, these are the methods that distinguish a functioning model from one that merely functions in theory. In the Dotsquares development group focused on creating custom LLMs for clients, this is the approach followed each and every time their AI solution must transition from prototype to a production-level application. Here’s how it’s done.
The raw size and computational demands of the model have increased drastically in recent years. The contemporary large language models may even exceed hundreds of billions of parameters. Surveys of model compression literature show that model compression and optimization methods have evolved into an essential branch of artificial intelligence engineering practice, rather than a complex topic for specialists. Mid-size models used for business purposes, such as searching, classifying, recommending, or generating text, may still be quite costly and inefficient to use without optimizations.
And there are very real practical ramifications as well; poor API response time, cloud invoices eating into ROI, and technical needs which preclude deployment on the edge. It has been demonstrated again and again in research studies that under 100 ms latency is the tipping point for successful enterprise AI implementations, users with slower responses tune out, and results are impacted. Correct implementation of AI model optimization methods is what determines whether the model will be feasible or not.
The good news? The main AI model optimization techniques, quantisation, pruning, knowledge distillation and inference optimisation, are well understood and battle-tested across a wide range of model types and deployment environments. Let's check each one.
Quantization of models involves lowering the precision of numbers used to describe the model weights from 32-bit floating point (FP32) to 16-bit (FP16) or 8-bit integers (INT8). This leads to a much smaller model with improved performance and decreased memory usage at the expense of some accuracy which can be minimized using quantization.
These are some impressive numbers. With quantization, there is about a 75% decrease in size of the model, while the speed is 2-4 times faster. The TensorRT INT8 quantization on ResNet-50 led to an accuracy loss of less than 1% while reducing the latency by 71%. As for BERT, the use of PyTorch dynamic quantization resulted in 75% decrease in model size and about 2x increase in speed, with almost no accuracy loss.
Model pruning eliminates unnecessary weights or neurons or entire layers that do not contribute much to the output of the model. Neural networks have a lot of extra parameters; there are a lot of weights whose contribution is negligible in terms of accuracy and hence can be either set to zero or pruned altogether.
Structured sparsity of NVIDIA (with 2:4 sparsity pattern) doubles the throughput on the A100 GPU. When used for BERT, sparsity of 90% reduced the size of the model from 420 MB to 42 MB while still retaining good accuracy. Pruning as little as 30% of the parameters can give 25% improvement in latency on mobile devices.
Knowledge distillation trains a smaller "student" model to replicate the behaviour of a larger "teacher" model, transferring the teacher's learned representations rather than its full parameter set. The result is a compact model that performs close to the original but costs a fraction as much to run.
The three methods mentioned above are not necessarily mutually exclusive. There are many cases where an LLM is pruned, quantised, and distilled in sequence to generate a more performant model. This sequence and balancing is the part that needs LLM engineering expertise.
Even after a model has been compressed, how it runs at inference time matters enormously. AI inference optimisation covers the techniques that make the deployment pipeline itself more efficient, independent of model size.
Key approaches include:
Cost perspective: Google's BERT deployment for search, after optimisation, achieved a 10x model size reduction, sub-10ms CPU latency and approximately 75% lower serving cost compared to the unoptimised version. Inference optimisation at scale isn't just a performance choice; it's a commercial one.
We have read about how different algorithms of compression and inference operate. Naturally, the next question would be: what about accuracy? A simple answer would be that there are trade-offs, and handling them is the whole point of the matter.
As for Quantisation-aware Training (QAT), it is one of the best methods to maintain accuracy while using quantisation – during the training process, the model gets trained already quantized, allowing it to adapt to the lower precision of the weights. It is always superior to post-training quantization in terms of maintaining accuracy in sensitive tasks.
As for pruning, the structured approach works better when it comes to maintaining accuracy due to the removal of whole units (heads, layers, channels), not just some weights across the model. As for the LLM fine-tuning methods, which are used to fine-tune the compressed model on domain-specific data to regain accuracy – this is where the accuracy is recovered.
Continuous evaluation matters here. Maintaining a consistent test set and running regular benchmarks before and after each optimisation step is the only reliable way to know what a compression technique has actually cost in terms of accuracy, and whether fine-tuning has recovered it.
Let's look into this more practically, because the right combination of techniques depends on what the deployment actually needs to do.
The honest reality is that most production deployments benefit from some combination of all these techniques; the balance is what requires careful testing and domain knowledge. Choosing a right parter as LLM engineering helps businesses set realistic expectations before they commit to a deployment architecture.
Optimisation doesn't end when the model is compressed. A few things consistently matter before a model goes live.
Benchmark on your actual data, not proxy datasets. Accuracy figures on standard benchmarks like GLUE or ImageNet don't necessarily translate to your specific use case. Run your optimised model against a representative sample of real production inputs before declaring it production-ready.
Plan for model drift and maintenance. An optimised model that was fine-tuned on data from 2024 will gradually drift in accuracy as the domain it operates in evolves. Build in a schedule for re-evaluation and, where needed, re-fine-tuning.
Test inference under load. A model that performs well on single-request benchmarks can behave very differently under concurrent request volumes. Load testing at expected production scale is essential, particularly for applications with spiky demand patterns.
Consider the full pipeline. Pre-processing, post-processing and data retrieval steps often account for a significant share of end-to-end latency. Optimising the model in isolation while ignoring the surrounding pipeline frequently leaves the biggest latency savings on the table.
Dotsquares approaches deployment as an end-to-end problem, not just compressing the model but designing the full custom AI development and inference pipeline around what the business actually needs to run reliably at scale.
What differentiates a working model from a production-ready model is the set of techniques for AI model optimisation. Techniques like quantisation, pruning and knowledge distillation shrink the size of the model, while inference optimisation, batching, caching, speculative decoding, optimised compilation for particular hardware make the deployment pipeline fast. All of them taken together, allow for achieving significant improvements in latency, costs and memory without compromising the model accuracy.
To get those results, it is important to approach optimisation as an engineering task. It requires rigorous benchmarking, load testing and a balanced choice of techniques, which will work well for your specific deployment case. If you are running a business and need that kind of expertise for AI systems you develop, then Dotsquares can help you with that.
Optimising an AI model involves shrinking the model’s size, computation costs and latency while maintaining maximum accuracy. This involves techniques such as quantisation, pruning, knowledge distillation and inference pipeline improvement, which are implemented in order to ensure the viability of the AI models for use in production.
Quantisation involves lowering the numerical precision of the model weights from 32 bits to 8 bits integer values. This results in a 75% reduction in model size and increases speed by 2 to 4 times with a maximum of 1% loss in accuracy when done well. This technique is one of the most widely used and cost effective optimization methods out there.
Pruning involves getting rid of low importance weights or layers in the current model, thus making it smaller and faster. Knowledge distillation involves training another smaller student model to behave in the same way as the large teacher model. Both methods help in shrinking the model, but knowledge distillation gives a more deployable model than pruning does.
It does but usually optimised models maintain accuracy in most cases. There are two methods that are used to compensate the loss of accuracy in the compression process; namely quantisation aware training and fine tuning after compression. For instance, DistilBert maintains 97% of the accuracy of BERT using 40% less parameters.
AI inference optimization optimizes the performance of the model when the model is served, regardless of the size of the model. The methods of doing this are continuous batching, key-value caching, speculative decoding and hardware-specific compilation. When combined together, it results in a 2-5 times performance gain in LLMs.
Drupal 10 reaches end of life on 9 December 2026. Learn about security risks, Drupal 11 benefits, and the steps to upgrade.
Keep ReadingExplore the best AI model optimisation techniques to improve speed, accuracy and efficiency, reduce inference costs, and prepare models for reliable business deployment.
Keep ReadingLearn how to build an AI personal finance app, including key features, AI capabilities, tech stack, bank data integration, development costs and MVP planning.
Keep Reading