0048: Pair cameras at the size they're shown
On this page
Status: accepted
Context
Pair's camera was a small second video track, 320 × 240 at 15 frames a second (0042), and the Cameras window scaled every picture down to 160 × 120 before stretching it to the window pixel by pixel. Made bigger, the tiles turned blocky. A fixed bigger size would fix it at the cost of bandwidth and encoding for tiles nobody makes bigger. A guest who joined from a browser, on a phone too, could see cameras but not show theirs.
Decision
- Captured at up to 960 × 720, always 4:3. The helper captures a camera format near 1280 × 720 at 15 frames or more, keeps the middle 4:3 of the picture (a wide camera loses its sides, never squashed), and scales it to 960 × 720.
- Sent at the size it's shown. Pair has a connection, and so an encoder, per person. Each viewer notes how wide it shows each sender's tile, in pixels, and once that has held for 300 ms asks that sender for it over a
cameradata channel, rounded up to 320, 480, 640 or 960 wide. The sender scales that one connection's encoder to it (scale_resolution_down_by), caps its bitrate for the size (0.3 to 1.5 Mbit/s) and keeps the frame rate, lowering the size first when the link or processor is short. Until a viewer asks, it gets 640 wide; a closed Cameras window asks for the smallest. - Drawn smoothly. Tiles keep the picture as it came, cropped to 4:3, and the window scales it to each tile with libyuv's box filter, once per frame and size.
- Browsers send a camera. A browser guest can turn their camera on, choose it (front or back on a phone) and be asked to turn it on. Their browser encodes AV1 or VP9, the codecs Pair offers. A browser can't name its streams, so a camera is known by its place as well as by its stream's name: every side adds voice, then the screen, then the camera, so the camera is the second video section. The browser sets that section to send both ways from the start, camera on or not, because a call never renegotiates; turning the camera on puts a track in it. A phone held upright sends a tall picture; the Cameras window shows its middle 4:3.
Consequences
- Big tiles are sharp and small ones cost less than before (320 wide at 0.3 Mbit/s).
- A sender encodes the camera once per connection at each viewer's size, as it already did at one size; the camera's capture and conversion are bigger than before (960 × 720 instead of 320 × 240).
- A Short Leash from before this version never asks, so it gets 640 wide, and ignores the
camerachannel. - Browser guests still watch and talk without control, drawing or sharing a screen; the camera is the only thing they gain.
- Telling the camera by its section ties the order of the sections to the protocol: voice, screen, camera, on every side.
- A browser that can't encode AV1 or VP9 can't send its camera; the page then has no camera button. Phones encode VP9 in software, so a browser sends 640 wide until asked for more.