Aperio

– Making the hidden visible.

Aperio is an interactive media object where initially concealed content is revealed through simple hand movements in front of a canvas. A camera detects the position of the hands and translates them into a virtual force field that fluidly pushes aside the covering mosaic tiles. This allows users to uncover the "hidden" backgrounds behind an image. Once fully revealed, auditory information about the displayed topic is provided.

Table of Contents

• Aperio
  • Table of Contents
  • Setup
  • Controls
  • Demonstration video
  • 1. Project Timeline & Milestones
  • 2. Requirements & Schematic Conditions
    • 2.1 Starting Points & Concept
    • 2.2 How Does the User Interact?
    • 2.3 Key Components
  • 3. Interaction Flow Diagram
  • 4. Hardware
  • 5. Prototype Approaches
    • 5.1 Functional Prototype
    • 5.2 Look-Like Prototype
  • 6. Result & Outlook
  • 7. Storyboard
  • 8. 3D Enclosure Model
  • LICENSE

Setup

Install some Python libraries and run the Python script:

git clone https://forgejo.hostlab.tech/suitdeer/aperio.git
sudo apt update
sudo apt install python3-opencv python3-picamera2 python3-vlc
cd aperio
python3 aperio.py

Autostart the program:

mkdir /home/gero/.config/autostart
tee /home/gero/.config/autostart/aperio.desktop <<EOF
[Desktop Entry]
Type=Application
Name=Clock
Exec=/usr/bin/python3 /home/gero/aperio/aperio.py
EOF

The following website was used for text-to-voice:

https://luvvoice.com

• German (austria) - Ingrid (Female)
• English (United States) - Jenny (Female)

---

Controls

• Press "R" on the keyboard to recalibrate. During calibration, you see four dots in each corner. Hold your hand over the green dot. During that click the left mouse button. Repeat for each corner to calibrate camera aspect ratio with beamer aspect ratio.

• Press "Q" to exit the aperio.py script

---

Demonstration video

---

1. Project Timeline & Milestones

---
config:
  theme: base
  themeVariables:
    background: "#1e1e1e"

    # Text
    textColor: "#707070"

    # Task bars
    taskBkgColor: "#666666"
    taskBorderColor: "#7a7a7a"
    taskTextColor: "#ffffff"

    # Sections
    sectionBkgColor: "#2a2a2a"
    sectionBkgColor2: "#242424"
    sectionTextColor: "#ffffff"

    # Grid
    gridColor: "#505050"

    # Axis
    lineColor: "#707070"

    # Title
    titleColor: "#707070"
---

gantt
    title Project Plan
    todayMarker off
    dateFormat YYYY-MM-DD
    axisFormat KW%W

    section Ideas
    Idea gathering                                  :a1, 2026-01-05, 7d
    Storyboard                                      :a2, after a1, 5d
    Order hardware                                  :a3, after a2, 7d

    section Hand Tracking
    Calibration logic                               :b1, after a3, 7d
    Hand detection                                  :b2, after b1, 7d
    Physics & repulsion                             :b3, after b2, 14d

    section Enclosure
    Modify E27 socket                               :c1, after a3, 2d
    Model and print enclosure                       :c2, after c1, 14d

    section Experience
    Background/media layer                          :d1, after b3, 3d
    Tile animation                                  :d2, after d1, 5d

    section Finalization
    Tuning                                          :e1, after d2, 2d
    Documentation                                   :e2, after e1, 7d
    Presentation                                    :e3, after e2, 5d

---

2. Requirements & Schematic Conditions

2.1 Starting Points & Concept

The basic idea is an interactive canvas (screen/projection) that is initially covered by a "mosaic" (i.e., an overlaying image that is split into tiles). A person interacts with this object physically in front of the canvas/screen:

• The hand position is captured via camera (Picamera2).
• The object (a colored hand/marker) creates a "magnetic repulsive force field".
• By swiping and stroking, the covering tiles move away from the hand (pushed outward from the hand's center), revealing the background and texts.

2.2 How Does the User Interact?

• The person approaches the screen/canvas.
• As the user moves their hands over the image, they push the foreground tile tiles away with invisible force.
• Experience: The interaction feels magical, fluid, and physically accurate (due to implemented inertia/friction). The user "digs" into a hidden narrative (image).

2.3 Key Components

1. The Camera Eye: Visually captures movements (filtered by specific colors in HSV space, e.g., red gloves/marker).
2. The Calibration (Homography): Mathematically transforms the skewed/distorted camera image into exact X/Y coordinates of the high-resolution screen.
3. The Grid (Tiles): A raster of image elements (approx. 80x80 pixels) that has an inertia vector and a home position.
4. The Trigger System: If the user pushes all tiles out of the image border, the next content (next image + text) is unlocked.

---

3. Interaction Flow Diagram

The following diagram shows the program flow and the user's interaction possibilities from startup to the media loop:

flowchart TD
    Start([Program Start: Init Camera, Window, Speaker & VLC]) --> AutoLoad{Saved calibration\n& sizes match?}
    
    AutoLoad -- Yes --> MainLoop[Main Interaction Loop]
    AutoLoad -- No --> Calib[Calibration Mode Active]
    
    Calib --> Click[User clicks on screen target]
    Click --> Collect[Collect hand position samples via HSV color]
    Collect --> Enough{Enough samples\nor timeout?}
    Enough -- No --> Collect
    Enough -- Yes --> Avg[Average samples -> camera point]
    Avg --> NextTgt{All 4 targets\ndone?}
    NextTgt -- No --> Click
    NextTgt -- Yes --> Compute[Compute Homography via RANSAC & save]
    
    Compute --> MainLoop[Main Interaction Loop]
    MainLoop --> ReadFrame[Capture frame: HSV mask AND MOG2 background subtraction]
    
    ReadFrame --> CTA{Idle > 15s?\nShow Call-to-Action animation}
    CTA --> Filter[Filter out false detections near Call-to-Action icon]
    Filter --> HandFound{Hand detected?}
    HandFound -- No --> ReturnRule{Tile idle > 10s?}
    
    HandFound -- Yes --> CalcPhys[Physics: repulsion from hand centers + contour collision bounce]
    CalcPhys --> ForceOut{>20% tiles out\n& not all out?}
    ForceOut -- Yes --> ForceOutPush[force_out: push remaining tiles outward]
    ForceOut -- No --> UpdatePos
    ForceOutPush --> UpdatePos[Tiles move via velocity & friction]
    
    ReturnRule -- Yes --> ReturnHome[Tiles return linearly to origin]
    ReturnRule -- No --> UpdatePos
    ReturnHome --> UpdatePos
    
    UpdatePos --> OutCheck{All tiles out of screen?}
    
    OutCheck -- Yes --> NextFg[Load next foreground image & play audio]
    OutCheck -- No --> HomeCheck{All tiles home?}
    
    NextFg --> allOutSet[all_were_out = true]
    allOutSet --> Render[Render background, tiles, contours & Call-to-Action]
    
    HomeCheck -- Yes --> NextBg[Stop audio & load next background image]
    HomeCheck -- No --> Render
    NextBg --> allHomeSet[all_were_home = true]
    allHomeSet --> Render
    
    Render --> Show[Display Canvas in Fullscreen]
    
    Show --> Inputs[Keyboard Inputs]
    Inputs -- q --> End([Exit Program])
    Inputs -- r --> ResetCalib[Delete calibration file & reset]
    Inputs -- None --> MainLoop
    
    ResetCalib --> Calib

---

4. Hardware

• Raspberry Pi 5 (2GB): https://www.raspberrypi.com/products/raspberry-pi-5/
• Raspberry Pi Camera Module 3: https://www.raspberrypi.com/products/camera-module-3/
• Raspberry Pi Active Cooler: https://www.raspberrypi.com/products/active-cooler/
• EWA Wireless Mini Speaker A106 Pro: https://www.ewaspeaker.com/collections/series-mini/products/ewa-a106-bluetooth-speaker
• Micro Projector: (Philips PicoPix 4010)

---

5. Prototype Approaches

5.1 Functional Prototype

This prototype currently exists in the Python script (aperio.py). It uses simple white contours and placeholder tiles to test the repulsion algorithm (and reveal hidden graphics). The focus of the functional prototype is not primarily on aesthetics. The initial questions are:

• Does the HSV color detection work (is my hand tracked accurately)?
• Is the latency (delay) between hand movement and tile movement real-time?
• Is the collision detection mathematics correct when swiping quickly across tiles?

5.2 Look-Like Prototype

In a next step, a dummy monitor/canvas (e.g., made of foam board or wood frame) should be built to illustrate how the user will ultimately stand in front of the art object. This serves as a mock-up to convey materiality without having complex technology integrated behind it. It helps to clarify formal questions such as:

• Where is the camera aesthetically best placed? Top or bottom?
• What is the ambient lighting in the final exhibition?
• Should the interaction be on a table-like flat grid or mounted on a wall?

6. Result & Outlook

The result combines hardware, mathematics (computer vision), and playful UI/UX into an organic discovery medium. The iterative development (from simple contour tracking to the evasion matrix and finally image replacement) is modularly organized, so that design content can be exchanged independently of tracking code.

---

7. Storyboard

---

8. 3D Enclosure Model

• Beamerhuelle-Vorne.glb
• Beamerhuelle-Hinten.glb

LICENSE

All contents of this project are protected by the AGPL3.0 LICENSE