Sep 4 2019
Whether for a drone or a car, creating image processing software for embedded systems requires plenty of time and is very costly.
Drone with a stereo camera. The small white box holds the embedded system, which evaluates the slightly offset images from the two cameras in real-time in order to detect obstacles. (Image credit: Fraunhofer IOSB)
At present, the Tulipp platform makes it feasible to develop energy-efficient embedded image processing systems more rapidly and less expensively, with a radical reduction in time-to-market.
The Fraunhofer Institute for Optronics, System Technologies, and Image Exploitation IOSB is a member of the EU consortium which streamlined the process.
At initial glance, driver assistance systems, drones, and mobile medical diagnostic equipment do not seem to have a lot in common. But in truth, they do: they all make increasing use of image processing components, for instance, for sensing pedestrians and obstacles. Image processing can also be used with mobile X-ray equipment to ensure sufficient image quality at lesser radiation levels, thus significantly decreasing radioactive exposure.
Conversely, a workstation computer, where energy consumption and dimensions are not predominantly crucial factors, applications like these necessitate lightweight, small, energy-efficient image processing components that are nonetheless real-time capable.
Hardware platforms based on traditional computer architectures and processors cannot appropriately meet these necessities. This is why embedded systems using field-programmable gate arrays (FPGAs) are frequently employed.
Field-programmable gate arrays are logic components having a circuit structure that can be easily configured using a special type of programing, typically involving the low-level language VHDL. But there is a hitch: Most of the image processing applications are written in higher-level programming languages such as C/C++, and their migration to the embedded systems is very complex.
Not only does VHDL vary significantly from other programming languages, but the code must also be modified to the particular hardware. This means even current VHDL programs cannot be shifted to other hardware. Software developers have to begin almost from scratch with each new system.
Starter Kit for Rapidly Implementing Especially Energy-Efficient Embedded Systems
A group of eight partners from six countries, including the Fraunhofer Institute for Optronics, System Technologies and Image Exploitation IOSB in Karlsruhe, has now significantly streamlined this procedure in the Tulipp project.
The result is a development platform consisting of design guidelines, a configurable hardware platform and a real-time-capable operating system that supports multicore processors, as well as a programing tool chain.
Dr -Ing. Igor Tchouchenkov, Group Manager, Fraunhofer IOSB
Tchouchenkov added, “A starter kit put on the market by one of our partners in Tulipp provides additional support. The starter kit makes developing such applications much faster and easier. Porting C++ programs to FPGA, which frequently means several months of work for the developer, can be handled within only a few weeks using the Tulipp starter kit.”
This means the developer first has to contemplate, based on the software programmed in C++, which code elements should be distributed to which hardware components and which program steps could be enhanced or parallelized. The formulated design procedures provide help with this job.
Then the starter kit is brought into play. It has the configurable hardware to which the required sensors and output devices can be linked, the multiprocessor-capable real-time operating system, and the so-called STHEM toolchain.
The applications in the STHEM toolchain make it possible to enhance the C++ program in such a way that it can be ported to the FPGA as easily and rapidly as possible.
One special focus of the toolchain is on energy optimization: after all, the aim is to design image processing systems that can be powered by a small battery whenever possible. The toolchain makes it possible to individually display and optimize energy consumption for each code function.
Dr.-Ing. Igor Tchouchenkov, Group Manager, Fraunhofer IOSB
One Hundred Times Faster than a High-End PC
The group looked at three particular use cases so as to develop and test the Tulipp platform: The Fraunhofer IOSB study team resolved stereo camera-based obstacle detection for drones, while other project partners looked into pedestrian detection in the surrounding area of a car and on live quality enhancement of X-ray images captured by mobile C-Arms during surgical operations. In the Tulipp platform, they ported the equivalent image processing software from C++ to FPGA.
The results are remarkable: The processing, which initially required several seconds to examine a single image on a high-end PC, can now work on the drone in real-time, that is, currently about 30 images are examined per second.
The speed of the pedestrian detection algorithm could be raised by a factor of 100: Presently, the system can examine 14 images per second compared to a single image every seven seconds. Improvement of X-ray image quality by applying noise-removing image filters helped in decreasing the intensity of radiation during surgical operations to one-fourth of the former level. Simultaneously, energy consumption could be considerably decreased for all three applications.