Streaming Video has Letterbox Effect

Symptom

Sometimes, the display of a streamed video is missing part of the edges, either at the left and right, or at the top and bottom, when the entire video is desired. Other times, there are black bars at the edges, either at the left and right, or at the top and bottom, when the entire display area is desired to be filled.  In other cases, the publisher preview may not match the subscribed video.

Applies To

• Streaming Video
• Vonage Video API

Resolution

When streaming video the stream is sent to both the publisher preview and to the Vonage Video Platform.  The local publisher view may be cropped to fit the UI design but the published stream will be the full stream from the video source.

To see the full width and height of the published video, even if it means having the letterbox effect, you may make use of the scale behaviour or fit mode of the publisher and subscriber objects.

The following example uses iOS (Swift) but this is available on other platforms.

On iOS you can use the OTVideoViewScaleBehavior property, and set it to OTVideoViewScaleBehaviorFit or OTVideoViewScaleBehaviorFill.

You do this when adding the Subscriber or Publisher to your View.

  subscriber?.viewScaleBehavior = .fit      // <=== Inserted line.
  subsView.frame = CGRect(x: 0, y: kWidgetHeight, width: kWidgetWidth, height: kWidgetHeight)
  view.addSubview(subsView)

  publisher.viewScaleBehavior = .fit
  pubView.frame = CGRect(x: 0, y: 0, width: kWidgetWidth, height: kWidgetHeight)
  view.addSubview(pubView)

Other Platforms

Similar scaling behaviours can be obtained on the Android SDK by using setStyle on either the Publisher or Subscriber.

To control scaling in the JS SDK, the CSS object-fit property could be used on a <video> container. To achieve a fit or fill scaling like we do for iOS and Android above, the values "contain" or "cover" could be used respectively.

Options

.fit (iOS) and contain (JS) = These will ensure the entire stream is displayed even if that means letterboxing the view.

.fill (iOS) and cover (JS) = These will ensure the container is "filled"/"covered" completely even if this means trimming the top and bottom or left and right of the video.

Cause

This issue occurs when a publisher shares the screen of a device with subscribers on devices with screens or UI layouts with different aspect ratios compared to the publishing device.

Due to the different aspect ratios, the rendered video on subscribers are sometimes missing the left and right edges or the top and bottom edges, or have the pillarbox or letterbox effect.

This can also happen when switching video quality.  640x480 streams have a 4:3 ratio but a 720p HD stream has a 16:9 ratio.  This will change how the video is rendered for both the publisher and the subscriber.