How Unicorn Scale Company Detect Millions of Violated Ecommerce Product Everyday

Cover Photo by James Lee on Unsplash

At Unicorn Scale Company, ensuring compliance with marketplace policies is a critical challenge. Sellers often attempt to upload prohibited items like drugs, animals, and sex toys by slightly modifying images to evade detection. Traditional hash-based matching (MD5, SHA) fails because even minor changes in an image result in completely different hashes. To solve this, we implemented perceptual hashing (pHash), a technique that identifies visually similar images even when altered. This article details our approach, implementation, and validation process.

Unicorn Scale Company processes around 3 to 5 million new product listings per day, making it essential to have an efficient and scalable solution for detecting policy violations.

But this regulations is usually different from each countries, in my country (Indonesia) for example, the government forbid us to sell sex toys, illegal drugs and others, some of the product often got passed from the evaluations, for example that we can see here:

Both images depict products containing illegal drugs and sex toys, which are prohibited from being sold.

Why Perceptual Hashing?

Unlike cryptographic hashes, which change drastically with small modifications, perceptual hashing produces similar hashes for visually similar images. This allows us to:

• Detect duplicate or near-duplicate images.
• Identify sellers re-uploading prohibited items with slight modifications.
• Improve enforcement of policy violations without relying on metadata or text descriptions.
• Detect bad actor sellers who often upload similar product images to evade detection. This was a recurring challenge on our platform, where policy-violating products were frequently re-listed with minor alterations.

System Architecture

Below is a simplified architecture diagram illustrating how our product violation detection system works and integrates with the product service as its backbone.

Workflow Overview:

Ops Team Workflow

A1. Operations teams establish new P-hash rules based on observations: Our dedicated team stays updated on the latest tactics used by bad actors.

A2. Requests are forwarded to the policy service by GCLB.

A3. P-hash list is updated with specific actions: The operations team can either upload an image or select a product from our admin dashboard to remove policy-violating products and register the corresponding P-hash image in the policy service.

Product Detection Workflow

B1. Seller Creates or Updates a Product: We process up to 5 million new product listings daily, and updates can reach 10 to 15 million products per day.

B2. Requests are forwarded to the product service by GCLB.

B3. Product data is saved to the product database: Any changes to a product are stored in the product database.

B4. Product service publishes an event: Product updates trigger events in our pub-sub topic, which other microservices listen to.

B5. Policy evaluation: The policy service consumer retrieves messages from the pub-sub system.

B6. P-hash conversion: All product images are converted into P-hash format by the policy service.

B7. Search for similar P-hashes: We search for similar P-hashes in our database using XOR and Hamming distance techniques. For example, we execute the following query:

SELECT image_id,
       bit_count(perceptual_hash # <actual_query_hash_value>) AS hamming_distance
FROM image_hashes
ORDER BY hamming_distance ASC
LIMIT 10;

B8. Take action based on P-hash rule: If a product’s P-hash matches the threshold for Hamming distance, we take specific actions based on the saved rules.

This architecture guarantees fast and scalable image moderation.

Practical Implementation in Golang

Now that we've covered the architecture, let's explore a practical implementation of perceptual hashing in Golang. We'll walk through the step-by-step process of converting an image into a pHash and comparing it with others, demonstrating how perceptual hashing functions in real-world applications.

Since the original product might be inappropriate for dev.to, I'll use a picture of a cute cat instead. 🐈🐈😺

I'll provide a high-level overview of how it works without delving too deeply into the low-level mechanics of pHash. For this demonstration, I'll be using a custom tool that I built. You can check out the package repository here.

Converting an Image into pHash

To generate a perceptual hash from an image, we first configure the hashing process with debugging enabled. This allows us to store intermediate visualizations for better analysis. Below is the updated implementation in Golang using the perceptualhash package:

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/insomnius/tools/perceptualhash"
)

func main() {
	// Enable debugging and set paths for intermediate images
	conf := perceptualhash.Config{
		Debug: true,
	}
	conf.DebugParameter.PreprocessedImagePath = filepath.Join("./debug", "preprocessed_cat.png")
	conf.DebugParameter.VisualizedImagePath = filepath.Join("./debug", "visualized_cat.png")

	// Ensure the debug directory exists
	if _, err := os.Stat("./debug"); os.IsNotExist(err) {
		os.Mkdir("./debug", 0755)
	}

	// Generate pHash for the given image
	hash, err := perceptualhash.FromPath("cat.png", conf)
	if err != nil {
		log.Fatal(err)
	}

	// Print the computed hash
	fmt.Println(hash)
}

• Configuration: Debugging is enabled to save intermediate images for inspection.
• Directory Handling: The script ensures the ./debug directory exists before writing debug images.
• Hash Generation: The FromPath function processes the image (cat.png) and generates a perceptual hash.
• Error Handling: If an issue occurs (e.g., file not found), the program logs an error and exits.
• Output: The computed perceptual hash is printed to the console.

Here are the output results, including the generated hash and debug images:

• Hash Output:

• Preprocessed Image (Scaled & Grayscale):

• Visualized pHash Representation: (the image is small so you need to zoom in)

These debug outputs help visualize how the image is transformed during the perceptual hashing process. 🚀

Next, let’s compare the original image with its altered versions—one that has been blurred, another that has been corrupted, and a third that has been resized.

Blurred Cat Image:

Corrupted Cat Image:

Resized Cat Image:

Despite these modifications, the generated perceptual hashes remain highly similar, with a Hamming distance score of just 1, demonstrating strong resilience to image alterations.

Comparison Output:

This highlights how perceptual hashing offers a cost-effective alternative to complex computer vision models. It enables robust similarity detection with minimal computational overhead, making it an efficient solution for policy enforcement in large-scale platforms. 🚀

You can see the complete example of this script in this github repository.

Conclusion

By implementing perceptual hashing, Unicorn Scaled Company significantly improved its ability to detect policy-violating products based on images. Our approach:

✅ Detected sellers re-uploading banned products with modifications.

✅ Worked efficiently at scale with Golang and Kubernetes.

✅ Reduced manual review efforts for compliance teams.

This solution became a key part of fraud detection system, enforcing policies while keeping the marketplace safe.

What should I write next?

• Technical details on how pHash works.
• Combining pHash with computer vision for deeper analysis.
• Implementing the product detection workflow in Golang.

Which one you like to know more? Let’s discuss in the comment sections! 🚀