The VisionKit camera framework
Specific for embedded Linux systems, we have developed a machine vision proof camera framework running since 2006. We can proudly say that it is one of the most versatile and efficient Linux powered frameworks around. It consists of the following modules:
All sensor properties and configurations are detected at the start of the video server. This also supports multi head camera implementations (tristated bus) without the need to recompile the kernel or the software. Thus, various camera solutions can run with one single software.
- New: Pipelined JPEG encoder IP core
- 8-12 bit input data
- Support for Xilinx Spartan3e (L1, L2), Spartan6 (up to 150 Mhz pixel clock) and Lattice ECP3 (L2)
- L1: one tap, for monochrome or high latency buffered systems
- L2: two tap, recommended for colour high speed YUV422 encoding
- Documentation of JPEG encoder SoC
- Preprocessor IP cores: CottonPicken Engine (Debayer, Filter kernel, ...), imaging sensor realtime control SoC
- ppivideo: High performance buffer queue based kernel driver for Blackfin/uClinux
- videoserver: Multi threaded camera server: Pipelined, quasi-simultaneous acquisition, image processing and data transmission via netpp or standard UDP/TCP video streaming. The videoserver is portable to other architectures (i86, ARM, ...)
- Sensor access library, supporting full register access, see supported sensor list on the bottom
- display: Remote display server with various video display options
- The Camasutra remote control user interface. Originally developed for cameras (hinting the origin of its name) it can be customized to support any type of device
The important differences to the existing SoC v4l2 framework are, that all configuration and control of sensor devices is running in userspace via the netpp framework. Advantages:
- No requirement to opensource your specs or drivers!
- No bloated kernell
- No/Very little driver maintenance required when upgrading to newer kernels
The image processing capabilities of the videoserver are basically featured by a FIFO buffer queue. The user tells the kernel (preferrably via the v4l2 USERPTR method) where the video buffers are stored. A DMA process automatically fills the buffers with incoming video data and notifies the user when a buffer is ready. Then, a raw buffer is taken off the FIFO queue, data is processed and is optionally stuffed back onto the FIFO queue for transmission or further processing.
The entire architecture was designed such that no data loss can occur, which allows to employ the system for line scan applications. The used three stage FIFO scheme ('fresh', 'process', 'deliver') provides maximum performance while being most flexible. For extra performance or low latency applications, Realtime driver models and extensions are supported as well.
- MJPEG streaming via http
- MPEG streaming using ffmpeg libraries
- 2D barcode reader, 3-5 frames per second
- Line scan application
- High reliability point tracking and blob detection
Since based on the netpp framework, the VisionKit provides
- Platform independent scripting support: Remote control your entire camera pool using Python scripts
- Fast design of camera type specific user interfaces in XML which can be loaded by the Camasutra GUI control tool - at runtime
Here we collected some examples from real applications. For debugging, not all frames normally need to be sent to the user display. In some cases, network bandwidth is not sufficient to send out all images, therefore two frame rates are given:
|Algorithm||CPU load (%)||Resolution||Processing rate [fps]||Tx rate [fps]|
|CC labeling (Blob detect)||36||640x480||20||10|
|JPEG HW-encoding||2 (DMA)||752x480||250||25|
|Example application of Camasutra remote control tool and remote display server. The processing is entirely done on the camera.
The VisionKit supports the following commercially available cameras and modules:
|Camera System||Sensor modules
Further reference solutions (automotive/medical) and ports to other architectures (TI/OMAP) on request
Support for other, custom sensor modules can be implemented easily using XML descriptions provided by the netpp framework