When the Only API Is the Screen

Not everything you need to integrate with will talk to you. Some systems only show you what's happening. There's no feed, no webhook, no export—just a screen with the answer on it, updating in real time, meant for a person to look at.

For a long time that meant a human had to sit there and watch. Now I point a piece of software at the pixels and let it watch instead.

The Screen Is the Interface

When I run a live card-selling stream, the platform doesn't hand me an event every time something sells. It updates a bar on the screen. A sale is a change in a rectangle of pixels, and for a while the only thing that knew a sale had happened was me, watching.

So I built a monitor that watches the stream the way I would. It screen-captures a few specific zones about fifteen times a second and looks for two things: a sudden drop in brightness in one region, which is how the platform signals a win, and text in another, which is the title of whatever just sold. Reading that text is just OCR—the same optical character recognition your phone uses to grab a number off a photo.

That's the whole trick. The screen was always the interface. I just stopped being the one reading it.

Reading Pixels Is the Easy Part

Getting text off the screen is not where the work is. The work is everything that happens after, because pixels lie.

A stream is a chaotic thing. Lighting shifts, overlays pop up, the camera moves. If your software treats every brightness change as a sale, it'll fire celebrations at nothing all night. So the win detector doesn't use an absolute threshold—it calibrates a baseline and only reacts to a drop relative to that baseline, which survives the room getting brighter or darker.

The OCR has the same problem in a different key. It reads text that's close to right but rarely perfect, so I never trust an exact match. The title it reads gets fuzzy-matched against the list of things it could plausibly be—in this case a grid of F1 drivers—and the closest real answer wins. Reading is a suggestion. Matching is the decision.

And because it's all firing in real time—an overlay, a set of smart lights, a sound, all at once—every action is wrapped so that one failing can't take the others down, and a cooldown stops a fast run of sales from stacking ten celebrations on top of each other. Screen-reading is easy. Not acting like a lunatic on what you read is the job.

Sometimes the Object Is the Interface

The same idea shows up somewhere less obvious. When I list a trading card for sale, I don't type in the player, the year, the set, and the grade. I take a photo of the card and let a vision model read it.

There's no API for "what card is this." The information is printed on the object itself, the way the sale was printed on the screen. So I treat the photo as the input, let the model pull out the fields I actually need, and hand back a finished listing. Same move as the livestream: the answer was sitting there in plain sight, meant for human eyes, and the trick was getting software to look.

The receipts work this way too. A vendor emails me a confirmation built for a person to skim, and a model reads it and pulls out what sold and what it cost. In every case the data was never hidden—it just wasn't offered in a format a machine was supposed to consume. So I stopped waiting to be offered it.

The Rules for Reading the World

A few things make this reliable instead of a party trick.

Calibrate, don't hardcode. Anything you read off the real world drifts—brightness, angle, format. Compare against a baseline you measure, not a number you guessed.

Read loosely, decide strictly. Assume the read is a little wrong. Match it against the small set of answers it could really be, and let the closest real one win.

Isolate every action. The moment you act on what you see, you're firing off things that can fail. Wrap each one so a single failure stays contained, and rate-limit so a burst of input doesn't become a burst of chaos.

None of this is how integration is supposed to work. You're supposed to get an API, a schema, a clean event. But plenty of the world still only exists on a screen or printed on a thing, meant for a person to look at. Teach software to look, and most of that stops being off-limits.