The VisionKit camera framework

Specific for embedded Linux systems, we have developed a machine vision proof camera framework running since 2006. We can proudly say that it is one of the most versatile and efficient Linux powered frameworks around. It consists of the following modules:

1. New: Pipelined JPEG encoder IP core
   • 8-12 bit input data
   • Support for Xilinx Spartan3e (L1, L2), Spartan6 (up to 150 Mhz pixel clock) and Lattice ECP3 (L2)
   • L1: one tap, for monochrome or high latency buffered systems
   • L2: two tap, recommended for colour high speed YUV422 encoding
   • Documentation of JPEG encoder SoC on request
2. Preprocessor IP cores: CottonPicken Engine (Debayer, Filter kernel, ...), imaging sensor realtime control SoC
3. ppivideo: High performance buffer queue based kernel driver for Blackfin/uClinux
4. videoserver: Multi threaded camera server: Pipelined, quasi-simultaneous acquisition, image processing and data transmission via netpp or standard UDP/TCP video streaming. The videoserver is portable to other architectures (i86, ARM, ...)
5. Sensor access library, supporting full register access, see supported sensor list on the bottom
6. display: Remote display server with various video display options
7. The Camasutra remote control user interface. Originally developed for cameras (hinting the origin of its name) it can be customized to support any type of device

The important differences to the existing SoC v4l2 framework are, that all configuration and control of sensor devices is running in userspace via the netpp framework. Advantages:

• No requirement to opensource your specs or drivers!
• No bloated kernell
• No/Very little driver maintenance required when upgrading to newer kernels

The image processing capabilities of the videoserver are basically featured by a FIFO buffer queue. The user tells the kernel (preferrably via the v4l2 USERPTR method) where the video buffers are stored. A DMA process automatically fills the buffers with incoming video data and notifies the user when a buffer is ready. Then, a raw buffer is taken off the FIFO queue, data is processed and is optionally stuffed back onto the FIFO queue for transmission or further processing.

The entire architecture was designed such that no data loss can occur, which allows to employ the system for line scan applications. The used three stage FIFO scheme ('fresh', 'process', 'deliver') provides maximum performance while being most flexible. For extra performance or low latency applications, Realtime driver models and extensions are supported as well.

Reference applications:

• MJPEG streaming via http
• MPEG streaming using ffmpeg libraries
• 2D barcode reader, 3-5 frames per second
• Line scan application
• High reliability point tracking and blob detection

Since based on the netpp framework, the VisionKit provides

• Platform independent scripting support: Remote control your entire camera pool using Python scripts
• Fast design of camera type specific user interfaces in XML which can be loaded by the Camasutra GUI control tool - at runtime

Screen shots

Performance examples Here we collected some examples from real applications. For debugging, not all frames normally need to be sent to the user display. In some cases, network bandwidth is not sufficient to send out all images, therefore two frame rates are given: Algorithm CPU load (%) Resolution Processing rate [fps] Tx rate [fps] CC labeling (Blob detect) 36 640x480 20 10 FlixTracking 67 176x176 124 31 JPEG SW-encoding 84 640x480 18 18 JPEG HW-encoding 2 (DMA) 752x480 250 25

Example application of Camasutra remote control tool and remote display server. The processing is entirely done on the camera.

Further documentation:

• Preliminary API documentation
• Example HTML property reference (This is a reference of all properties and registers supported by the standard Visionkit supporting a few Aptina sensors)

The VisionKit supports the following commercially available cameras and modules:

Support for other, custom sensor modules can be implemented easily using XML descriptions provided by the netpp framework