Behind the Scenes: The Technology That Powers ProtectionAI
- Aug 9
- 7 min read
Thank you, as always, for supporting PAPS and its work.
This time, we would like to share a look behind the scenes at the technology behind ProtectionAI, a system PAPS is developing.
To keep searching for images of victims, we must continuously collect enormous numbers of images and videos from the internet, detect faces, and match them—without interruption.
This part of our work is rarely visible, but keeping ProtectionAI running actually depends on many layers of technical ingenuity.
■ What is ProtectionAI?
Private sexual images and videos taken or shared online without consent (NCII) are often reposted on other sites and pages even after being removed once. ProtectionAI is a system that, based on facial photos provided by survivors who wish to use it, converts facial features into numerical data, searches the internet for images and videos that may show that person, and connects the findings to takedown requests.
This requires continuously reviewing a huge number of images and videos, detecting whether faces appear in them, and matching their features.
Keeping this "continuous search" running is what enables ongoing recovery for those who have been harmed.
■ "Paprika", the system that collects images and videos
In a previous newsletter, we introduced Paprika, the collection system PAPS is developing. Paprika is a "crawler"—a system that automatically visits pages on the internet and collects images and videos. Development has progressed, and its final form is finally coming into view.
Currently, it collects data by running roughly 300 Chrome browsers at all times on about 150 virtual machines (VMs). However, every website is structured differently.
It is not unusual for a method that easily retrieves images on one site to fail completely on another.
That is why Paprika uses a mechanism called the "Code-Gen Loop".
In this mechanism, an AI known as an LLM writes a program, runs it, checks the result, and automatically repeats the process of fixing it if it does not work. ChatGPT is one example of a service built on LLMs.
For example, if images or videos cannot be retrieved from a website, the AI writes code in the Python programming language. If the code fails when executed, the AI rewrites it based on the results and keeps improving it until retrieval succeeds.
That said, running the AI every time we check a single web page would be enormously expensive. So once a program retrieves data successfully, it is saved as a "skill" that can be reused on the same website. The next time we examine another page on that site, there is no need to have the AI write a program from scratch.
By separating the parts where the AI thinks from the parts where learned methods are reused, we can crawl a large number of pages efficiently.
■ Facial feature data now exceeds 80 million records
The facial feature data used for matching in ProtectionAI has surpassed 80 million records. That amounts to 880,000 images per day (roughly 600 images per minute of training).
Each piece of data serves as a clue for finding the images and videos we need. At the same time, images collected from the internet include not only photographs but also illustrations. The face detection system sometimes judges illustrations of non-existent people to be "human faces", mixing unnecessary data into the matching process.
The more of this "noise" there is, the heavier the burden on the processes that search for the data we actually need.
We are therefore also developing a dedicated AI model to automatically identify and remove this noise.
■ Behind the scenes (1): Finding faces in videos is very "heavy" work
Among all of ProtectionAI's processing, video is especially demanding.
A still image only requires analyzing a single picture, but a video is made up of a huge number of consecutive still frames.
To find faces in a single video, we must analyze many of the frames that make it up, one after another.
If we relied on CPUs alone, even high-performance CPUs would hit their limits after analyzing just a few videos simultaneously.
So we brought in GPUs, which excel at image processing and performing massive numbers of calculations in parallel.
With GPUs, we can process videos and images far more efficiently than with CPUs alone. But simply adding more GPUs does not solve everything.
To keep feeding large volumes of data to the GPUs, we needed to speed up both "where the data is stored" and "how the data is delivered".
■ Behind the scenes (2): Overcoming the SSD "wall" with RAM
ProtectionAI reads and writes very large amounts of data at high speed.
As a result, with consumer SSDs (storage drives), durability—not just speed—became a problem. In our environment, sustained heavy writes sometimes exhausted an SSD's write endurance in about six months. And if reads and writes to the SSD cannot keep up, the overall system slows down no matter how much headroom the CPUs and GPUs have.
So we decided to run MinIO, the system that temporarily stores data during processing, on RAM instead of SSDs. RAM is what is usually called "memory" in a PC. It reads and writes data far faster than an SSD. Of course, data stored in RAM disappears when the power goes out. But what we keep there is temporary, in-process data: if needed, it can be regenerated from the original data, so it can be restored even if lost.
Furthermore, the roughly 300 Chrome browsers were also causing SSD slowdowns and shortening drive lifespans through heavy reads and writes. So we moved Chrome's temporary data to a "RAM disk"—a storage area that lives in RAM.
This improved read/write speeds and greatly reduced the load on our SSDs. The trade-off, of course, is that it requires a large amount of RAM. And right now, the cost of sourcing server memory is a major challenge: some 96GB DDR5 RDIMM modules we bought for about 100,000 yen a year ago now cost roughly nine times as much.
Buying everything new was simply not realistic. So we collected used server parts from around 2015—sometimes resorting to analog solutions like drilling holes with a drill press to adapt old parts to today's standards—and bought used DDR4 LRDIMMs on eBay and elsewhere, adding about 1TB of memory. Through these efforts, the system now runs on roughly 2.4TB of memory in total. Behind the glamour of AI technology, used servers and second-hand memory are quietly playing a starring role.
■ Behind the scenes (3): A 10-gigabit line that would not go fast?
Because ProtectionAI and Paprika collect huge volumes of images and videos, our network traffic grows every year. We upgraded to a fiber line of up to 10Gbps and updated our internal network and main servers to support 10Gbps as well. Yet in actual operation, speeds plateaued at around 2–3Gbps. Our monitoring system showed massive packet loss at the router.
A closer look revealed that the conversion processing for IPv4 traffic was concentrating load on just part of the router's CPU, which was constantly maxed out. So we decided to build our own router using VyOS, a Linux-based open-source router platform.
Having already spent a lot on an expensive router, we had little budget left. We gathered mostly used server parts on eBay for about 60,000 yen. It was not easy: we dug through tech blogs for the settings needed for our connection, and a serious bug in our configuration script surfaced right before the switchover. Still, we completed the migration at 4 a.m. with only about 30 minutes of downtime.
As a result, average throughput rose from about 1.14Gbps to 2.17Gbps, and packet loss, previously around 6.8%, dropped to nearly 0%.
Rather than simply "making the line faster", this cleared the "traffic jam" that had been forming at the router. In ProtectionAI, if any single component—CPU, GPU, SSD, RAM, or network—gets congested, the entire pipeline slows down.
By resolving these hard-to-see bottlenecks one by one, we are building an environment that can keep searching through more images and videos, reliably.
■ Behind the scenes (4): A custom pipeline that coordinates some ten machines
Even with many servers and GPUs, they cannot deliver their full performance if each works in isolation. We need a mechanism that directs the overall traffic: which server takes which job, and what runs next once a task finishes.
Coordinating multiple computers to manage processing this way is called "orchestration" (think of the conductor of an orchestra). Well-known distributed processing systems such as Ray exist, but when we tried them, they proved too heavyweight for our current use case and our scale of around ten machines, and did not quite fit. So we developed our own pipeline system tailored to ProtectionAI's scale and workloads.
It monitors the status of each server and GPU and directs traffic by handing the next task to whichever machine is free.
▼ Technical documentationhttps://paps-jp.github.io/pipeline/
Building this system also ran into one major problem.
Throughput would not stay constant—it "oscillated", speeding up and slowing down, and the GPUs sometimes stalled waiting for work. Even with high-performance GPUs, their computing power goes unused if the side feeding them work cannot keep up.
The idea we adopted resembles a car's gear shifting. The right gear differs between pulling away and cruising at high speed. Likewise, we introduced a mechanism that changes how work is allocated to the GPUs depending on the system's processing conditions.
As a result, we can now run the GPUs stably while smoothing out the processing waves, and continuously process more images and videos.
■ Technology in service of survivors
In developing ProtectionAI, we grapple not only with the AI model itself but with many other questions:
How do we collect enormous numbers of images and videos?
How can we analyze large volumes of video quickly?
How do we keep the hardware running without interruption?
These are the kinds of challenges we face.
Each improvement may seem modest on its own.
But together, they let us search more images and videos, faster and more continuously. And what lies beyond that is connecting people harmed by the non-consensual spread of sexual images and videos to support even a little sooner.
We will continue to share updates on ProtectionAI's development and the work behind it.
Thank you for your continued support of PAPS.
