May 11 2021
Skoltech scientists have created a new monitoring system for agricultural applications that performs real-time image segmentation on board the drone to identify hogweed. The research was published in a high-profile journal, IEEE Transactions on Computers.
Sosnovsky’s hogweed is equally hazardous for farming, local ecosystems, and human health. Direct contact with human skin, especially if aggravated by exposure to the Sun, causes severe burns that require continuous medical care and take weeks to heal. The rampant spread of Sosnovsky’s hogweed has become a real environmental disaster that extends across the whole of Russia, from its central part to Siberia and from Karelia to the Caucasus. Every year, the government allocates huge budgets (last year, 350 million rubles for Moscow alone) for hogweed elimination. Eradicating the poisonous plant has become one of the biggest challenges for Russian farming, the environment, and healthcare.
In the mid-20th century, there were plans to use hogweed as a fodder crop, given its quick growth, low maintenance requirements, and fast proliferation. However, it soon transpired that hogweed was no good as a livestock feed and its exceptional natural properties were a major problem rather than an advantage. One plant can produce up to 100,000 seeds per year, which are easily dispersed by the wind. This means that a single plant accidentally left behind makes the clearing operation utterly pointless.
Accurate real-time localization of hogweed was the first issue that researchers from the Skoltech Center for Computational and Data-Intensive Science and Engineering (CDISE) encountered when they started developing their monitoring platform two years ago. “Conventional monitoring methods are not effective enough, since the ground observations are highly dependent on the human factor, while space remote sensing can spot large thickets only. Satellite images do not have sufficient resolution to discern individual plants. Besides, traditional monitoring strongly depends on weather and satellite revisit periods and, therefore, cannot provide up-to-date information,” lead author and Skoltech PhD graduate Alexander Menshchikov explains.
The researchers decided to use drones that are able to capture up-to-date high-resolution hogweed images even in cloudy weather and opted for in-flight data acquisition and processing on board the drone instead of the classical “data capture – orthophotomap – data analysis” scheme. “Even though the the traditional approach provides exhaustive information about the area, it is nearly as effective as processing data on board with segmentation for one type of object, i.e. Sosnovsky’s hogweed. Besides, in the traditional method, after-flight image stitching and analysis takes several hours, whereas in-flight processing produces real-time data which are immediately downloaded to the base station, so that the clearing of the targeted area could start before the drone lands,” Alexander adds.
The new monitoring solution uses a drone and a compact on-board computer which runs “heavy” segmentation algorithms based on Fully Convolutional Neural Networks (FCNN) that can identify an irregularly shaped object (in this case, Sosnovsky’s hogweed) pixel-by-pixel. This will help to discern individual plants and raise the chances of killing all the weeds in the selected area.
Running FCNNs on low-power hardware, such as single-board computers (SBC), was the main hindrance for the project. Since there are only a limited number of computers that have sufficient resources and processors that support FCNN, the researchers had to find a suitable SBC architecture and optimize FCNN to make it run on the selected hardware version.
“We chose popular architectures, UNet, SegNet, and ResNet, for our neural networks and adapted them for the SBC. We installed and flight-tested our monitoring system on board the drone which covered an area of up to 28 hectares in 40 minutes, flying at an altitude of 10 meters. And it did not miss a single weed!” Skoltech assistant professor and project supervisor Andrey Somov comments.
“Our system displays a multifold increase in localization efficiency, even though it processes 4K images at a modest speed of 0.7 fps,” Andrey adds.
The new approach holds great promise for farming: it could be used to monitor other crops, identify various vegetative indicators, assess plant health, and detect plant diseases by applying multispectral imagery.
The hogweed project is a collaborative study involving MSc and PhD students and researchers of several Skoltech centers: CDISE, CDMM (Skoltech Center for Design, Manufacturing and Materials), and SSC (Skoltech Space Center): Alexander Menshchikov, Dmitry Shadrin, Viktor Prutyanov, Daniil Lopatkin, Sergey Sosnin, Evgeny Tsykunov, Evgeny Yakovlev, and Andrey Somov.