Using Orbbec 3D cameras (UVC)

Table of Contents

• • Introduction
  • Instructions
• Code
  • Code Explanation
    • Python
    • C++
  • Results
    • Python
    • C++
  • Note

Prev Tutorial: Using Orbbec Astra 3D cameras
Next Tutorial: Using Creative Senz3D and other Intel RealSense SDK compatible depth sensors

Original author	Jinyue Chen
Compatibility	OpenCV >= 4.10

Introduction

This tutorial is devoted to the Orbbec 3D cameras based on UVC protocol. For the use of the older Orbbec 3D cameras which depends on OpenNI, please refer to the previous tutorial.

Unlike working with the OpenNI based Astra 3D cameras which requires OpenCV built with OpenNI2 SDK, Orbbec SDK is not required to be installed for accessing Orbbec UVC 3D cameras via OpenCV. By using cv::VideoCapture class, users get the stream data from 3D cameras, similar to working with USB cameras. The calibration and alignment of the depth map and color image are done internally.

Instructions

In order to use the 3D cameras with OpenCV. You can refer to Get Started to install OpenCV.

Note since 4.11 on, Mac OS users need to compile OpenCV from source with flag -DOBSENSOR_USE_ORBBEC_SDK=ON in order to use the cameras:

cmake -DOBSENSOR_USE_ORBBEC_SDK=ON ..
make
sudo make install

By default, when -DOBSENSOR_USE_ORBBEC_SDK=ON is enabled, OrbbecSDK v2 is used (i.e., ORBBEC_SDK_VERSION defaults to 2); it supports the entire Orbbec Gemini 330 series.

If you need legacy cameras such as Orbbec Femto, Gemini2XL, or Astra+, switch to OrbbecSDK v1 with the flag -DORBBEC_SDK_VERSION=1:

cmake -DOBSENSOR_USE_ORBBEC_SDK=ON -DORBBEC_SDK_VERSION=1 ..
make -j
sudo make install

Code

Python

This tutorial code's is shown lines below. You can also download it from here

#!/usr/bin/env python
 
import numpy as np
import sys
import cv2 as cv
 
 
def main():
    # Open Orbbec depth sensor
    orbbec_cap = cv.VideoCapture(0, cv.CAP_OBSENSOR)
    if orbbec_cap.isOpened() == False:
        sys.exit("Fail to open camera.")
 
    while True:
        # Grab data from the camera
        if orbbec_cap.grab():
            # RGB data
            ret_bgr, bgr_image = orbbec_cap.retrieve(None, cv.CAP_OBSENSOR_BGR_IMAGE)
            if ret_bgr:
                cv.imshow("BGR", bgr_image)
 
            # depth data
            ret_depth, depth_map = orbbec_cap.retrieve(None, cv.CAP_OBSENSOR_DEPTH_MAP)
            if ret_depth:
                color_depth_map = cv.normalize(depth_map, None, 0, 255, cv.NORM_MINMAX, cv.CV_8UC1)
                color_depth_map = cv.applyColorMap(color_depth_map, cv.COLORMAP_JET)
                cv.imshow("DEPTH", color_depth_map)
        else:
            print("Fail to grab data from the camera.")
 
        if cv.pollKey() >= 0:
            break
 
    orbbec_cap.release()
 
 
if __name__ == '__main__':
    main()

C++

This tutorial code's is shown lines below. You can also download it from here

 
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <iostream>
 
using namespace cv;
int main(int argc, char** argv)
{
    cv::CommandLineParser parser(argc, argv,
                                 "{help h ? |  | help message}"
                                 "{dw |  | depth width }"
                                 "{dh |  | depth height }"
                                 "{df |  | depth fps }"
                                 "{cw |  | color width }"
                                 "{ch |  | color height }"
                                 "{cf |  | depth fps }"
 
    );
    if (parser.has("help"))
    {
        parser.printMessage();
        return 0;
    }
 
    std::vector<int> params;
    if (parser.has("dw"))
    {
        params.push_back(CAP_PROP_OBSENSOR_DEPTH_WIDTH);
        params.push_back(parser.get<int>("dw"));
    }
 
    if (parser.has("dh"))
    {
        params.push_back(CAP_PROP_OBSENSOR_DEPTH_HEIGHT);
        params.push_back(parser.get<int>("dh"));
    }
 
    if (parser.has("df"))
    {
        params.push_back(CAP_PROP_OBSENSOR_DEPTH_FPS);
        params.push_back(parser.get<int>("df"));
    }
 
    if (parser.has("cw"))
    {
        params.push_back(CAP_PROP_FRAME_WIDTH);
        params.push_back(parser.get<int>("cw"));
    }
 
    if (parser.has("ch"))
    {
        params.push_back(CAP_PROP_FRAME_HEIGHT);
        params.push_back(parser.get<int>("ch"));
    }
 
    if (parser.has("cf"))
    {
        params.push_back(CAP_PROP_FPS);
        params.push_back(parser.get<int>("cf"));
    }
 
    VideoCapture obsensorCapture;
 
    if (params.empty())
        obsensorCapture.open(0, CAP_OBSENSOR);
    else
        obsensorCapture.open(0, CAP_OBSENSOR, params);
    if(!obsensorCapture.isOpened()) {
        std::cerr << "Failed to open obsensor capture! Index out of range or no response from device";
        return -1;
    }
 
    double fx = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_FX);
    double fy = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_FY);
    double cx = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_CX);
    double cy = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_CY);
 
    double k1 = obsensorCapture.get(CAP_PROP_OBSENSOR_COLOR_DISTORTION_K1);
    double k2 = obsensorCapture.get(CAP_PROP_OBSENSOR_COLOR_DISTORTION_K2);
    double k3 = obsensorCapture.get(CAP_PROP_OBSENSOR_COLOR_DISTORTION_K3);
    double k4 = obsensorCapture.get(CAP_PROP_OBSENSOR_COLOR_DISTORTION_K4);
    double k5 = obsensorCapture.get(CAP_PROP_OBSENSOR_COLOR_DISTORTION_K5);
    double k6 = obsensorCapture.get(CAP_PROP_OBSENSOR_COLOR_DISTORTION_K6);
    double p1 = obsensorCapture.get(CAP_PROP_OBSENSOR_COLOR_DISTORTION_P1);
    double p2 = obsensorCapture.get(CAP_PROP_OBSENSOR_COLOR_DISTORTION_P1);
 
    std::cout << "obsensor camera intrinsic params: fx=" << fx << ", fy=" << fy << ", cx=" << cx << ", cy=" << cy << std::endl;
    std::cout << "obsensor camera distortion params: k,p=" << k1 << ", " << k2 << ", " << k3 << ", "
                                                           << k4 << ", " << k5 << ", " << k6 << ", "
                                                           << p1 << ", " << p2 << std::endl;
 
    Mat image;
    Mat depthMap;
    Mat adjDepthMap;
 
    // Minimum depth value
    const double minVal = 300;
    // Maximum depth value
    const double maxVal = 5000;
    while (true)
    {
        // Grab depth map like this:
        // obsensorCapture >> depthMap;
 
        // Another way to grab depth map (and bgr image).
        if (obsensorCapture.grab())
        {
            if (obsensorCapture.retrieve(image, CAP_OBSENSOR_BGR_IMAGE))
            {
                imshow("RGB", image);
            }
 
            if (obsensorCapture.retrieve(depthMap, CAP_OBSENSOR_DEPTH_MAP))
            {
                depthMap.convertTo(adjDepthMap, CV_8U, 255.0 / (maxVal - minVal), -minVal * 255.0 / (maxVal - minVal));
                applyColorMap(adjDepthMap, adjDepthMap, COLORMAP_JET);
                imshow("DEPTH", adjDepthMap);
            }
 
            // depth map overlay on bgr image
            static const float alpha = 0.6f;
            if (!image.empty() && !depthMap.empty())
            {
                depthMap.convertTo(adjDepthMap, CV_8U, 255.0 / (maxVal - minVal), -minVal * 255.0 / (maxVal - minVal));
                cv::resize(adjDepthMap, adjDepthMap, cv::Size(image.cols, image.rows));
                for (int i = 0; i < image.rows; i++)
                {
                    for (int j = 0; j < image.cols; j++)
                    {
                        cv::Vec3b& outRgb = image.at<cv::Vec3b>(i, j);
                        uint8_t depthValue = 255 - adjDepthMap.at<uint8_t>(i, j);
                        if (depthValue != 0 && depthValue != 255)
                        {
                            outRgb[0] = (uint8_t)(outRgb[0] * (1.0f - alpha) + depthValue * alpha);
                            outRgb[1] = (uint8_t)(outRgb[1] * (1.0f - alpha) + depthValue *  alpha);
                            outRgb[2] = (uint8_t)(outRgb[2] * (1.0f - alpha) + depthValue *  alpha);
                        }
                    }
                }
                imshow("DepthToColor", image);
            }
            image.release();
            depthMap.release();
        }
 
        if (pollKey() >= 0)
            break;
    }
    return 0;
}

Code Explanation

Python

• Open Orbbec Depth Sensor: Using cv.VideoCapture(0, cv.CAP_OBSENSOR) to attempt to open the first Orbbec depth sensor device. If the camera fails to open, the program will exit and display an error message.
• Loop to Grab and Process Data: In an infinite loop, the code continuously grabs data from the camera. The orbbec_cap.grab() method attempts to grab a frame.
• Process BGR Image: Using orbbec_cap.retrieve(None, cv.CAP_OBSENSOR_BGR_IMAGE) to retrieve the BGR image data. If successfully retrieved, the BGR image is displayed in a window using cv.imshow("BGR", bgr_image).
• Process Depth Image: Using orbbec_cap.retrieve(None, cv.CAP_OBSENSOR_DEPTH_MAP) to retrieve the depth image data. If successfully retrieved, the depth image is first normalized to a range of 0 to 255, then a false color image is applied, and the result is displayed in a window using cv.imshow("DEPTH", color_depth_map).
• Keyboard Interrupt: Using cv.pollKey() to detect keyboard events. If a key is pressed, the loop breaks and the program ends.
• Release Resources: After exiting the loop, the camera resources are released using orbbec_cap.release().

C++

• Open Orbbec Depth Sensor: Using VideoCapture obsensorCapture(0, CAP_OBSENSOR) to attempt to open the first Orbbec depth sensor device. If the camera fails to open, an error message is displayed, and the program exits.
• Retrieve Camera Intrinsic Parameters: Using obsensorCapture.get() to retrieve the intrinsic parameters of the camera, including focal lengths (fx, fy) and principal points (cx, cy).
• Loop to Grab and Process Data: In an infinite loop, the code continuously grabs data from the camera. The obsensorCapture.grab() method attempts to grab a frame.
• Process BGR Image: Using obsensorCapture.retrieve(image, CAP_OBSENSOR_BGR_IMAGE) to retrieve the BGR image data. If successfully retrieved, the BGR image is displayed in a window using imshow("BGR", image).
• Process Depth Image: Using obsensorCapture.retrieve(depthMap, CAP_OBSENSOR_DEPTH_MAP) to retrieve the depth image data. If successfully retrieved, the depth image is normalized and a false color image is applied, then the result is displayed in a window using imshow("DEPTH", adjDepthMap). The retrieved depth values are in millimeters and are truncated to a range between 300 and 5000 (millimeter). This fixed range can be interpreted as a truncation based on the depth camera's depth range, removing invalid pixels on the depth map.
• Overlay Depth Map on BGR Image: Convert the depth map to an 8-bit image, resize it to match the BGR image size, and overlay it on the BGR image with a specified transparency (alpha). The overlaid image is displayed in a window using imshow("DepthToColor", image).
• Keyboard Interrupt: Using pollKey() to detect keyboard events. If a key is pressed, the loop breaks and the program ends.
• Release Resources: After exiting the loop, the camera resources are released.

Results

Python

BGR And DEPTH frame

C++

BGR And DEPTH And DepthToColor frame

Note

• Mac users need sudo privileges to execute the code.
• Firmware: If you’re using an Orbbec UVC 3D camera, please ensure your camera’s firmware is updated to the latest version to avoid potential compatibility issues. For more details, see Orbbec’s Release Notes.