Teaching BuffBox to See: Building a Machine Vision Pipeline for Trading Cards

One of the more interesting engineering challenges I've taken on while building BuffBox is teaching an iPhone to recognize trading cards. The idea sounds simple enough: point your camera at a Magic: The Gathering card, take a picture, and have the app figure out what you're looking at. But getting that to work reliably involves quite a bit more engineering than I initially expected.

I've been building a machine learning and image processing pipeline that can find a card in a photograph, determine its boundaries, correct its orientation, and compare the resulting image against a catalog of known cards. The goal is to eventually support other trading card games using the same underlying technology. I don't want to train a completely new system every time we add another game.

The technology behind it is pretty cool, and I thought I'd share some of what I've been working on, the decisions I've made, and why I ended up taking a slightly more complicated path than simply downloading an existing model.

First, We Need to Find the Card

The first challenge is figuring out where a card is in an image. A photograph might contain a card sitting on a desk, surrounded by other objects, at an awkward angle, under imperfect lighting, or partially covered by reflections from a sleeve. Before BuffBox can identify the card, it needs to isolate it from everything else in the photograph.

For this, I'm using YOLOX-Tiny, a lightweight convolutional neural network (CNN) from the YOLO (You Only Look Once) family of object detection models. I'm fine-tuning an existing pretrained model using LibreYOLO, a permissively licensed computer vision framework. Rather than teaching the neural network to recognize individual Magic cards, I'm training it to recognize trading cards as a single class of objects.

That's an important distinction. The model has no idea what Serra Angel is, what edition it belongs to, or how much it's worth. It only needs to recognize that something in the image looks like a trading card and return a bounding box describing its location. The actual identification happens later in the pipeline.

I'm training the model using synthetic images, with the option to incorporate real-world photographs provided with consent. The synthetic data allows me to generate training scenes containing different card positions, orientations, backgrounds, and lighting conditions. I've also been collecting examples of unusual card treatments, including borderless, showcase, full-art, and extended-art printings, to increase the visual variety represented during training.

The training configuration is deliberately focused on the kinds of images BuffBox expects to encounter. I'm using a 416-pixel input resolution and limiting certain augmentations to better reflect the single-card scanning experience. The model still needs to tolerate cards at different angles, but I want the training process to reflect realistic photographs rather than creating unnecessarily complicated scenes.

One of the advantages of this approach is that the detection model doesn't need to be retrained every time Wizards of the Coast releases a new set. A newly printed card is still a trading card, regardless of whether the recognition catalog knows what it is.

Finding the Card Isn't the Same as Identifying It

Once YOLOX detects a card, we're only partway there. The detector returns an axis-aligned bounding box, which describes the general rectangular area containing the card. Unfortunately, if the card is rotated or photographed at an angle, that box may also contain a considerable amount of the table, playmat, or whatever happens to be behind it.

To handle this, I've built additional image processing around the detector. BuffBox evaluates the detected region to make sure it resembles a card, looking at characteristics such as its geometry, texture, and edge structure. From there, the application can attempt to recover the card's four corners and use that geometry to produce a more accurately aligned image.

This is where things get interesting. The neural network is useful for finding an object, but conventional image processing can handle much of the work needed to prepare that object for recognition. I don't need another neural network for every step, and I like being able to solve problems with relatively simple algorithms where they make sense.

The pipeline also uses different confidence thresholds depending on what the application is doing. For example, the live camera preview can show a possible detection before the application is sufficiently confident to automatically capture it. The idea is to make the scanning experience responsive without allowing every uncertain detection to trigger a capture. I'm still tuning these values against real-world photographs, and they'll likely change as testing continues.

Once BuffBox has a usable image of the card, we can finally attempt to determine what it is.

Identifying Cards Without Training a Giant Classifier

Magic: The Gathering has an enormous catalog, with many cards appearing across multiple sets, printings, and visual treatments. I could attempt to train an image classification model to recognize individual cards, but that would create an ongoing maintenance problem. Every new release would potentially require changes to the classification model, and adding another trading card game would multiply the work.

Instead, BuffBox uses perceptual hashing.

A perceptual hash, or pHash, produces a compact fingerprint of an image based on its visual characteristics. Unlike a conventional cryptographic hash, which changes dramatically when the underlying data changes, a perceptual hash is designed so that visually similar images produce similar fingerprints.

I'm using a discrete cosine transform (DCT) to convert a normalized image into a frequency-domain representation. The lower-frequency components capture much of the image's overall visual structure, giving us a relatively compact way to describe what the card looks like.

In the current implementation, the image is normalized to 32 × 32 pixels and the top-left 8 × 8 DCT coefficients are examined to construct a 64-bit fingerprint.

Once we have that fingerprint, comparing it against known cards is remarkably straightforward. We use Hamming distance, which counts how many bits differ between two values. By applying an XOR operation to the fingerprints and counting the differing bits, BuffBox can quickly determine which reference images are the most visually similar to the scanned card.

Of course, the surrounding pipeline is considerably more complicated. Getting consistent results requires careful image normalization, and photographs don't always arrive in the correct orientation. BuffBox generates perceptual hashes for several orientations and compares them against the reference catalog to find the closest match.

I've implemented the hashing algorithm in both Swift and Go, with the intention of using a compatible fingerprint format across the mobile application and backend. Maintaining consistent image preprocessing and DCT behavior across the two implementations is important, so cross-platform validation is part of the work involved.

This approach allows the application to narrow down possible card matches without requiring the neural network to know anything about the card's identity.

When Two Cards Look Almost Identical

Perceptual hashing has an interesting limitation: sometimes it's very good at finding visually similar images, even when those images represent different printings of a card.

Magic has plenty of examples where the same artwork appears across different sets. A perceptual hash might identify several of those printings as close matches, even though a collector would consider them distinct cards.

That's where Optical Character Recognition (OCR) comes into play.

Once the image-matching process has narrowed the possibilities, BuffBox can use Apple's Vision framework to read text from the card. I'm currently focusing on the title region, with support for checking an alternative region on layouts where the name isn't in its usual position.

I've found that the more accurate recognition mode is worth using for final identification. Faster OCR can occasionally return convincing nonsense, particularly when dealing with glare, small text, or photographs of cards displayed on screens.

The goal is to use information from multiple stages to improve recognition. The visual fingerprint identifies likely candidates, and the text provides another piece of evidence. There are still challenges with nearly identical printings, unusual frames, and imperfect photographs, but the architecture gives me several ways to improve accuracy without replacing the entire system.

Getting Everything Running on an iPhone

One of my priorities has been making the detection process run directly on the user's device. Sending every camera frame to a server for machine learning inference would introduce network latency, increase operating costs, and create an unnecessary dependency on a hosted service.

Instead, I'm converting the trained model into Apple's Core ML format and using the Vision framework to perform detection locally. That allows the application to take advantage of the computing hardware available on supported Apple devices without requiring server-side inference for the detection stage.

The conversion process turned out to be more involved than simply exporting a model.

The exported YOLOX model needed additional work to interact correctly with Apple's Vision framework. In particular, the model's non-maximum suppression (NMS) outputs needed suitable class labels and normalized coordinates. I built an export script that examines the Core ML model, adjusts its output pipeline where necessary, and validates the resulting model before packaging it for the application.

I've also added benchmarking tools to measure detection accuracy and inference latency. My current target is to keep on-device detection below 50 milliseconds at the 95th percentile, although that's a performance objective rather than a claim that every device already achieves it.

The nice thing about local inference is that it supports one of my broader goals for BuffBox: keeping core collection management functionality free without creating an expensive infrastructure dependency for every scan.

Why I Chose LibreYOLO Instead of Ultralytics

There's another part of this project worth discussing, particularly for developers thinking about incorporating machine learning into their own applications.

When I started evaluating the available YOLO implementations, Ultralytics was an obvious candidate. It's a popular platform with mature training and deployment tooling, and I have a lot of respect for the engineering behind it. Unfortunately, its licensing model presented complications for the kind of application I wanted to build.

Ultralytics offers its software under the AGPL-3.0 license, alongside a commercial Enterprise licensing option. The AGPL allows commercial activity, but it includes source-code disclosure obligations that can apply when covered software is incorporated into and distributed as part of a larger application. Ultralytics also publishes licensing guidance concerning applications and models built using its technology.

For BuffBox, I wasn't comfortable taking on those obligations without fully committing to an AGPL-compatible distribution model. The alternative was pursuing a commercial license before the project had generated any revenue.

Neither option was especially appealing for a one-person development shop.

Ultimately, I chose LibreYOLO, which uses the MIT license for its framework, together with a YOLOX-Tiny model. This provides a more permissive foundation for building a proprietary mobile application, although the exact terms for pretrained weights and training data still need to be respected.

That doesn't mean I wrote YOLOX from scratch. The model architecture and framework are products of existing open-source work, and those projects deserve credit. My contribution is in adapting the model to trading-card detection, developing the training and evaluation workflow, implementing the surrounding image processing and recognition systems, and getting everything working together inside BuffBox.

The choice has required additional engineering, especially around Core ML export and integration, but I'm comfortable with the tradeoff. It gives the project more flexibility over how its software and model weights are distributed.

I don't have a problem with Ultralytics choosing the licensing terms it believes are appropriate. Open-source maintainers have every right to decide how their work is licensed, just as developers building applications need to decide which dependencies make sense for their projects.

Why the Business Side Matters

I've said before that I intend to keep BuffBox's core functionality free, and that's still the plan. I believe collectors should have access to good collection management tools without being charged a subscription simply to organize cards they've already purchased.

That said, I'm also trying to build something sustainable.

Machine learning training consumes computing resources, and maintaining an application takes time. There are costs associated with development, infrastructure, updates, and supporting the community. Running inference locally helps keep those expenses manageable, but it doesn't eliminate them entirely.

I'm not building BuffBox because I expect to make an enormous amount of money. I'd genuinely be happy to see the project eventually cover its own costs and provide enough revenue to support continued development. If it grows beyond that, wonderful!

But I do need to preserve the ability to charge fairly for services that introduce recurring costs. Otherwise, the project could eventually become too expensive to maintain, which wouldn't be good for me or the people depending on it.

That's one reason I pay attention to software licensing before committing to a particular technology. I want to understand the obligations associated with the tools I'm using and make sure they don't conflict with the long-term plans for BuffBox.

Will I Open-Source the Pipeline?

I'm seriously considering it.

The more I work on the machine vision system, the more I realize that much of it could be useful outside BuffBox. The detection model doesn't fundamentally care whether it's looking at a Magic card, a Pokémon card, or something else with similar physical characteristics. The surrounding image processing and identification techniques could also be adapted for other applications.

I'd like to eventually release some of this work as a standalone project, including the training scripts, detection tooling, and potentially the trained weights. Before doing that, I want to clean up the implementation, improve documentation, establish reliable benchmarks, and review the licensing and redistribution rights associated with the training materials.

I'm particularly interested in making the detector useful independently of the BuffBox application. I think there's value in sharing a tool that someone else can integrate into their own project without requiring an account or access to my infrastructure.

I've benefited from open-source software throughout my career, and I'd like to contribute something useful back. If another developer can improve the model, add support for another type of collectible, or find an application I hadn't considered, that would be pretty exciting.

Nothing is finalized yet, but it's something I intend to keep exploring.

Where Things Stand Today

The machine vision pipeline is still a work in progress. I'm continuing to improve detection accuracy, expand the training data, test unusual card treatments, and refine the image processing that happens between detecting a card and identifying it.

There's also a lot of work happening elsewhere in BuffBox, including collection management, deck building, and Match Mode. The card recognition pipeline is only one piece of the larger application, but it's been one of the most technically challenging and enjoyable pieces to develop.

Eventually, I want to support additional trading card games without having to rebuild the recognition system every time. That's a major reason I've taken a modular approach rather than training one enormous classifier to recognize a fixed catalog of Magic cards.

I realize this has been a fairly technical update compared to some of my other posts, but I enjoy sharing how things work, and I hope at least a few of you find the engineering as interesting as I do.

If you're working with computer vision, machine learning, or mobile development, I'd love to hear what you're building. And if you're a collector, I'm looking forward to getting BuffBox into your hands and seeing how it performs with real collections.

Thanks for following along!

Brent
Developer, BuffBox