Transforming Airline Security with AR-Based Trolley Inspection

Airport security has a problem that doesn't get talked about much outside the industry, catering and trolley inspection. Every trolley that goes onto a flight, food, beverages, service items, needs to be checked before it's cleared, and for years that's meant someone physically going through each tray by hand. It's slow, it's repetitive, and it leaves room for the kind of human error that shows up exactly when you can't afford it.

We took this on as a proof-of-concept project to answer a specific question: could augmented reality and AI actually replace, or at least dramatically speed up, that manual process without sacrificing accuracy? Here's how we built it, what got in the way, and what the results looked like.

Why Aviation Security Is a Different Kind of Challenge

Aviation security sits in a category of its own. The margin for error is close to zero, the volume of items to check is high, and every process still has to move fast enough to keep flights on schedule. That combination made it exactly the kind of problem worth solving with real engineering, not just automation for its own sake.

What We Set Out to Build

The goal was straightforward to state and genuinely hard to build: create a system using AR/VR app development that lets security personnel scan food and beverage trolleys using AR headsets and get real-time, AI-generated information overlaid directly onto what they're looking at. 

Instead of manually lifting trays and checking each item, the idea was for a security officer to simply look at a trolley through an AR headset and have the system identify trays, food items, and any potentially dangerous objects, think knives, blades, or anything resembling a weapon, instantly, with a color-coded alert showing whether something needed a closer look.

Under the hood, the system runs on computer vision powered by the Faster R-CNN algorithm for object detection, a Python and FastAPI backend, PostgreSQL for storing scan data and generating reports, and AWS handling the real-time processing load. The aim wasn't just detection, it was detection fast enough and accurate enough to actually fit into a real security workflow.

Why Object Detection at This Level Is Harder Than It Sounds

Building this wasn't a matter of plugging in an off-the-shelf detection model and calling it done. A handful of genuinely tough problems showed up early, and each one needed its own solution.

Data Collection and Dataset Creation

There simply wasn't a ready-made dataset of airline trolley items anywhere. No public dataset covers trays, catering items, and security-relevant objects in this exact context. That meant collecting image data manually from scratch, then expanding it through augmentation techniques like rotation and brightness adjustment so the model could generalise beyond the exact photos we'd taken.

Accuracy of Object Detection

Detection accuracy is only as good as the conditions it's tested under, and trolley environments are inconsistent by nature. Cabin lighting shifts, camera angles vary depending on how a security officer holds the headset, and item placement inside a trolley is never perfectly uniform. Getting reliable detection across all of that required extensive training and repeated optimisation, not a single training run and done.

Multi-Level Detection Complexity

Here's a challenge that isn't obvious until you're deep in it, the system doesn't just need to detect a tray, it needs to detect what's inside the tray too, which is a fundamentally different visual problem. A single model trying to do both at once kept underperforming on one task or the other. The fix was splitting it into two dedicated models, one purely for tray detection, and a second focused specifically on identifying items and flagging dangerous objects within them.

Real-Time Processing and Performance

Detection that's accurate but slow isn't useful in a live security setting, officers need results the moment they look at a trolley, not several seconds later. Getting real-time inference working meant optimising how data moved between the AR headset, the backend, and the detection models themselves, balancing processing speed against the accuracy the models needed to stay reliable.

AR Device Limitations and Stability

AR hardware is only as good as the conditions it's used in. Camera clarity, stable positioning, and correct alignment all directly affect detection quality. Any movement, blur, or inconsistent lighting risked incorrect readings or lag between what the officer sees and what the system reports, which meant device stability had to be treated as seriously as the AI model itself.

The Solution: Building an Optimised Detection Framework

Solving each of these problems meant combining data strategy, model architecture decisions, and backend optimisation into a single, coherent system, using custom AI development solutions rather than treating them as separate fixes bolted together. 

• Dataset creation and augmentation came first. The custom dataset built from manually collected trolley, tray, and item images was expanded significantly through rotation, flipping, and brightness augmentation, giving the model a much broader range of conditions to learn from than the original raw images alone could offer.
• Improving detection accuracy meant training across that diverse, augmented dataset specifically covering different lighting conditions and camera angles, with Faster R-CNN handling the core object detection task to get precision to a level that could actually be trusted in a security context.
• The multi-model detection approach solved the tray-versus-item complexity directly. Running two purpose-built models, one for trays, one for items and dangerous objects, rather than forcing a single model to do both, made a measurable difference in how reliably each detection task performed.
• Real-time processing optimisation came down to FastAPI and AWS working together to handle live data efficiently, paired with model-level optimisation to cut down processing delay without giving up detection accuracy in the process.
• AR stability and alignment improvements were handled through practical guidelines, proper lighting conditions, stable headset positioning, and correct alignment protocols, all built into how the system expects to be used, which reduced scanning errors considerably.

Put together, this gave us a genuinely unified detection framework, one that handles the AI side and the AR side as one connected system rather than two separate technologies stitched together after the fact.

The Impact: A Smarter, Faster Approach to Airline Security

The finished system changed what trolley inspection actually looks like in practice. Automation, real-time detection, and AR-assisted decision-making replaced a process that used to depend entirely on manual checking, and the difference showed up in several concrete ways.

• Faster inspection process. AR-enabled scanning lets security personnel move through trolleys considerably faster than manual inspection ever allowed, with real-time detection cutting out the delays that come with checking every item by hand.
• Improved accuracy and safety. Faster R-CNN-based detection identifies trays, food items, and dangerous objects with a level of precision manual checks can't consistently match, reducing the human error that naturally creeps into repetitive inspection work.
• Real-time visual assistance. Color-coded alerts, green, amber, red, appear directly in the AR interface, giving officers an immediate, unambiguous read on item status instead of a judgment call made under time pressure.
• Centralised monitoring and reporting. Every scan, alert, and report gets stored in a centralised database, accessible through an admin dashboard, which means security teams get real tracking and analysis capability instead of paper logs or scattered records.
• Enhanced operational efficiency. With detection automated, security personnel spend less time on manual checking and more time on the judgment calls and decision-making that actually need a human in the loop.

What This Project Demonstrates

This system was built to prove a point, that AR and AI aren't just buzzwords when applied to a real operational problem, they can genuinely transform a slow, manual security process into something faster, more accurate, and easier to monitor at scale. It's a strong example of what's possible when computer vision, cloud infrastructure, and wearable AR hardware are engineered to work together as one system rather than treated as separate add-ons.

If your business is sitting on a manual, repetitive process that feels like it should be smarter by now, this is exactly the kind of problem our team enjoys solving. Get in touch with Dotsquares to talk through what an AR or AI-driven solution could look like for your operations.