Building an AR and AI-Powered Trolley Inspection System for Airline Security

Anyone who's watched a ground crew load a food and beverage trolley before a flight knows how much of it still comes down to a person eyeballing trays and hoping nothing's been missed. It's slow; it depends entirely on how alert that one person is at that one moment, and on a busy turnaround, "thorough" and "fast" don't always sit well together. That tension is exactly what pushed one of our aviation clients to ask whether augmented reality and computer vision could take over the parts of trolley inspection that shouldn't need a human eye at all.

What came out of that conversation is a system built entirely in-house by our AI/ML development team: an AR headset that lets security staff scan a trolley and instantly see, overlaid right on the tray in front of them, whether everything inside it is safe to fly. No client name attached to this one, but the build itself is worth walking through, because it's a good example of what's actually possible when AI and AR are paired for a real operational problem rather than a demo.

Why Manual Trolley Checks Weren't Cutting It

The brief was straightforward on paper: reduce how long trolley inspections take, and reduce how often something gets missed. In practice, neither of those is simple to solve with software, because the problem sits at the intersection of a few genuinely hard technical challenges.

There was no existing dataset for airline trolley items, so nothing to train a model on straight out of the box. Every tray photo, every food item, every object that needed flagging had to be collected and labelled from scratch. Lighting on an aircraft, camera angle, how items happened to be arranged in a tray, all of that affected how reliably a model could tell a butter knife from a dinner fork. On top of that, the system had to detect two different things at once: the trays themselves, and then whatever was sitting inside them, which meant one detection model wasn't going to be enough. And because the whole point was speed, results had to come back in real time, on an AR headset, without lag that would make security staff trust the tech less than the manual check it was replacing.

None of these are unusual problems in AI/ML development. What made this one demanding was solving all of them together, inside a device that has to work reliably in a moving, imperfect physical environment.

How the System Was Built

Starting With a Dataset That Didn't Exist

The first real hurdle was data, or the lack of it. Our team manually collected images of trolleys, trays, and the items typically found inside them, then used augmentation techniques like rotation and brightness adjustment to stretch that dataset further, so the model would still perform under lighting conditions it hadn't seen an exact copy of before.

Getting Detection Accuracy to a Usable Level

Detection accuracy is where a lot of computer vision projects quietly fall apart, and this one had the same risk. Camera angle, lighting, and how objects were positioned inside a tray all threw off early results. The model was trained across a deliberately varied set of conditions using Faster R-CNN, which gave the accuracy needed to tell a genuinely dangerous object from something that just looked similar in a low-resolution frame.

Splitting the Problem Into Two Models

Trying to detect both a tray and everything inside it with a single model turned out to be the wrong approach; it asked one system to do two fairly different jobs at once. The fix was to split it: one model dedicated to identifying trays, and a second focused purely on the items and any potentially dangerous objects inside them, like knives or firearms. Running them together gave a more reliable read than a single combined model ever managed.

Making It Work in Real Time

An inspection tool that lags is a tool nobody trusts. The backend runs on Python and FastAPI, with AWS handling the processing load, and a fair amount of the optimisation work went into trimming response time down to something that actually feels instant on the headset, not just fast on a benchmark chart.

Keeping the AR Side Stable

AR headsets are only as good as their stability. A shaky hand, poor lighting, or a slightly off angle can throw off a scan just as easily as a bad model can. Getting this right meant setting practical guidelines for positioning and lighting, rather than assuming the hardware would compensate on its own.

What the System Actually Delivers

Security personnel wear the headset, scan a trolley, and get a colour-coded read almost immediately: green for clear, amber for worth a second look, red for stop and check. Every scan and every alert gets logged to a PostgreSQL database and shows up on an admin dashboard, so there's a full history to refer back to rather than a result that disappears the moment the scan's done.

The practical upside is fairly direct. Inspections move faster because the system is doing the first pass instead of a person doing all of it manually. Accuracy improves because the two-model setup is genuinely better at telling safe items from concerning ones than a rushed visual check tends to be. And because everything's centrally logged, there's an actual audit trail if a flagged item or a scan result ever needs to be reviewed later.

None of this replaces the judgement of trained security staff, and it isn't meant to. What it does is take the repetitive, error-prone part of the job, staring at tray after tray under time pressure, and hand it to a system that doesn't get tired on the fifteenth trolley of the shift the way a person naturally does.

What This Says About Where AI and AR Are Heading in Aviation

Aviation security has generally been slower than other industries to bring AI into day-to-day operations, mostly because the margin for error is so much lower. Projects like this one suggest that's starting to shift, not because AI is replacing security judgement, but because it's getting good enough to reliably handle the first-pass detection work that used to eat up so much manual time.

For any team weighing up whether a custom build like this makes sense, the honest answer is that it depends less on the AI model and more on getting the fundamentals right, the dataset, the real-time backend, and a device experience people will actually trust in the field. That's the part that separates a working AI ML development company from one that can only ship a proof of concept. If you're looking into ai & ml development solutions for a similarly specific operational problem, this is roughly the shape the work tends to take: less about the algorithm being clever, and more about it being reliable enough to hand real responsibility to