Debajyoti Karmaker

The goal of my research is to develop intelligent guidance systems for Unmanned Aerial Vehicles (UAVs) by investigating how birds use their vision, decision making capability, and agile maneuverability to avoid collisions with obstacles while flying in complex environments. My research interests are in computer vision, machine learning, and deep learning. I am particularly interested in the areas of object detection, pose estimation, action recognition, and biologically inspired collision avoidance strategies for quadcopters. More specifically, I have been designing and developing efficient algorithms for the detection and tracking of highly deformable objects (like birds), and uncovering novel guidance laws and algorithms that birds use to avoid mid-air collisions, with the ultimate aim of applying some of these principles to the design of collision avoidance systems for UAVs.

As a researcher, I am focused on developing computer vision algorithms for object detection and tracking. Ultimately, the algorithms have to be useful in real-world applications, thus, empirical validation of the proposed algorithm is necessary. As a result, my research outcomes generally include creation of new models and algorithms that are immediately applicable to multiple domains. Following this philosophy, I have developed models and computer algorithms for reliable automated detection of birds in videos and still images, automatic identification and localization of their parts (head, tail, wingtips), and used these algorithms to detect and track birds in free flight to investigate how they avoid mid-air collisions. My research so far has pursued the following goals:

OBJECT DETECTION AND TRACKING

Object detection and tracking are active and important research areas in computer vision. Object detection and recognition are applied in many areas of computer vision and other disciplines, including image retrieval, security, surveillance, automated vehicle systems and machine inspection.

Fine-grained object recognition: Flying birds are one of the more challenging ‘objects’ to detect, partly because of their constantly changing shape. To address this challenge, I introduced WHOG [c], a new framework that can support fine-grained recognition of birds. This is a challenging problem due to the high diversity of poses that a bird can assume, and the high variation of plumage color and texture. I was able to achieve a major improvement in precision over well-known detection techniques like DPM, which are compromised by high pose variation. I provided a strategy which uses bottom-up pose clustering, to deal with extreme pose variations. I also introduced and demonstrated the value of a new global feature descriptor (WHoG) that is well suited for the detection of objects with diverse textures. Furthermore, by combining pose clustering and scale invariant color features i.e. combining global features with local features, I constructed a powerful detector that is robust to background clutter and internal body textures or stripes. In principle, this method can be applied to the detection of not just birds, but any highly articulated object. This work [c] was published at ICARCV, 2016.

Detection and tracking of small, poorly lit, deformable objects: The problem formulation in [c] relies on sufficient temporal and spatial resolution of the bird images, which may not be always available - especially during long-term tracking of small flying creatures such as birds or insects with a stationary camera, when the object is far away from the camera. In this research [b], I proposed a set of simple algorithms for detecting and tracking multiple, small, poorly lit, deformable objects in environments that feature drastic changes in background and foreground illumination. The proposed detection method is capable of successfully dealing with a variety of rapid background changes, without the need for continuous updating of a background model. This detection method - based on the decomposition of a logical map - is associated with reduced complexity and computational expense. This work [b] was published at ISVC, 2018.

Image Denoising: The need for efficient image restoration methods has grown with the use of high-speed video cameras, as the exposure time for high-speed cameras is constrained by the frame rate, which limits the SNR. In this research [a], I demonstrated a denoising technique which improves the performance of the detection algorithm described [b]. The image denoising technique described here is based on a dynamic filtering process where a dynamic mask is oriented to match the local gradient, with a weighting profile that is proportional to the magnitude profile of the local gradient. This work [a] was published at ROBIO, 2018.

a) Debajyoti Karmaker, Ingo Schiffner, Michael Wilson, Mandyam V Srinivasan. “Image denoising with Weighted ORientation-Matched Filters (WORM)” in International Conference on Robotics and Biomimetics (ROBIO), 2018.

b) Debajyoti Karmaker, Ingo Schiffner, Michael Wilson, Mandyam V Srinivasan. “The bird gets caught by the WORM: tracking multiple deformable objects in noisy environments using Weight ORdered logic Maps” in International Symposium on Visual Computing (ISVC), 2018.

c) Debajyoti Karmaker, Ingo Schiffner, Reuben Strydom, Mandyam V Srinivasan. “WHoG: A weighted HoG-based scheme for the detection of birds and identification of their poses in natural environments” in ICARCV 2016.

ACTIVITY RECOGNITION IN VIDEOS

Currently, the keyboard and the mouse are the primary interfaces between man and computer. Progressively, more technology is emerging for communication with computers. Many of these new devices are for special-purpose applications and are expensive. The promise of computer vision for human-computer interaction (HCI) is great and vision-based interfaces would allow the capture and recording of unencumbered, large-scale spatial motion that can discover and define new forms of interaction.

With this in mind, we proposed a Simple Human Computer Interaction Machine that uses a Physical Gesture Framework, implemented on conventional equipment such as a low resolution camera, that offers a number of applications in the gaming industry and in smart houses. The main goal of these researchs [b][c] is to outline a mechanism of computer vision for controlling any application or hardware. In this research [c], we demonstrated how a generalized framework (SHIMPG) [c] can be seamlessly integrated with any networked controlled application, as it works as a separate engine which can be easily integrated. Additionally, we developed a hand gesture recognition algorithm and demonstrated successful integration with SHIMPG. This work was published at ICARCV, 2014. In another study, we proposed a methodology for real-time hand detection and tracking, with stereo camera calibration and disparity mapping. We demonstrated how depth information can be used together with the Convex Hull algorithm to improve human hand detection and tracking. This work [a] was published at ICAEE, 2015.

a) Md. Farhad Zaman, Samma Tasnim Mossarrat, Fahad Islam, and Debajyoti Karmaker. in “International Conference on Advances in Electrical Engineering (ICAEE)” 2015.

b) Debajyoti Karmaker, AZM Ehtesham Chowdhury, Md. Saef Ullah Miah, Md. Al Imran, and Md. Hafizur Rahman. “Cricket Shot Classification Using Motion Vector” In International Conference on Computing Technology and Information Management (ICCTIM), 2015.

c) Md. Asif Ur Rahman, Md. Saef Ullah Miah, M. Abrar Fahad, and Debajyoti Karmaker. “SHIMPG: Simple human interaction with machine using Physical Gesture” in International Conference on Control Automation Robotics & Vision (ICARCV), 2014.

BIO-INSPIRED COLLISION AVOIDANCE STRATEGIES FOR UAVS 

There are more aircraft occupying our skies than ever before. The inexorable increase of air traffic, particularly of UAVs, underscores the increasing need for inexpensive and reliable systems for the automatic avoidance of mid-air collisions. The aviation industry is now undergoing another revolution. Services like Amazon Prime Air, which use UAVs for product delivery are likely to operate near the ground. When operating in these conditions, unmanned aircraft need to avoid collisions with the natural environment (e.g. terrain and trees) and the built environment (e.g. power lines, towers, buildings etc.). Many unmanned aircraft use maps to assist navigation close to the ground. Maps, however, go out of date. Also, to be effective, these maps need to be detailed, and three-dimensional. What is required is an independent, self-reliant means of navigating close to the ground that does not depend upon external information or infrastructure.

Birds have been navigating the Earth's environment for millions of years. They are extremely agile fliers, traveling rapidly through dense natural environments and complex urban landscapes without colliding with obstacles or each other. Nature has seemingly already solved the very same problems we are currently facing in the design of UAVs. While much work has been done on long range navigation in birds, little attention has been devoted to short-range navigation, and to investigate how birds achieve feats such as obstacle avoidance, which is clearly important for an intelligent UAV guidance system. To address this problem, we collaborated Boeing Research & Technology Australia Ltd to investigate how birds avoid mid-air collisions, to understand how they identify obstacles and evaluate their threat levels, and to uncover the strategies that they use to avoid collisions with them.

Behavioral Analysis of Bird Flight Trajectories: In order to investigate the collision avoidance strategies in birds, it is important to gain insight into the basic aspects of flight behavior so that we can better analyze the collision avoidance techniques. To study this, I investigated the paths taken by birds when they fly in a tunnel whose cross section is large enough to permit a variety of trajectories while moving from one end of the tunnel to the other. Firstly, do the birds exhibit a preferred flight path while flying in a tunnel? If so, does this preference persist with the passage of time? Secondly, do all birds use the same flight trajectory, or does the preferred trajectory vary from bird to bird? Thirdly, if an obstacle is placed in a bird’s preferred path, does it switch to an entirely different flight path, or does it try to retain its originally preferred flight path by making just a brief detour around the obstacle?

The results revealed that individual birds display distinct flight trajectories that were consistent from flight to flight. I then investigated the robustness of each bird's trajectory by interposing a disk-shaped obstacle in its preferred flight path. Each bird continued to fly along its preferred flight path up to a point very close to the obstacle, before veering away to avoid a collision and subsequently returning to its preferred path. The last-minute avoidance maneuver exhibited by the birds suggests that collisions are avoided by restricting the magnitude of the optic flow generated by the obstacle to a maximum value of about 700 deg/sec. Thus, birds show a high propensity to stick to their individual, preferred flight paths [a]. Each bird tends to maintain its preferred flight path even when it is confronted with a clearly visible obstacle, and does not adopt a different, safer route.

 Strategies for Avoiding Mid-air Collisions: In this research [b], I investigated whether, and, if so, how, birds use vision to avoid a moving obstacle. Budgerigars, (Melopsittacus undulates), were filmed in 3D using high-speed video cameras as they flew along a 25 m tunnel in which a swinging pendulum, carrying a 41 cm disc, presented an oscillating obstacle that moved back and forth across the bird's flight path. To my knowledge, mine is the first study to examine how birds cope with a moving obstacle. The collision-avoidance scenario investigated here is very different from the well-studied phenomena of birds flying in a flock, or aircraft flying in a formation. There, all of the birds (or aircraft) are flying predominantly in the same direction, making imminent collisions easier to detect and prevent. This particular scenario deals with a single bird that is suddenly confronted by a moving obstacle, say, a tree branch or another bird, approaching from a completely different direction. The results reveal that birds approaching the moving pendulum ensure a collision-free passage by always aiming for the visually wider gap at the time of crossing the pendulum. Furthermore, they enhance the safety of the passage

through the wider gap by increasing their flight speed slightly when the pendulum is approaching them, and decreasing it slightly when it is receding, thus ensuring that the chosen gap is as wide as possible at the time of transit.

However, the challenge posed to the birds in [a,b], looked at how birds avoid a diskshaped obstacle that is stationary or moving from side to side like a pendulum. To further investigate how birds treat obstacles that present different levels of threat, and the strategies that they employ to avoid them, the birds were presented with a mechanical bird whose wings are either flapping, or stationary. The results suggest that distance to the mechanical bird is gauged using cues based on optic flow when the wings are stationary, and cues most likely

based on image size when the wings are flapping. Birds avoided the static mechanical bird by restricting the magnitude of the optic flow of the obstacle to a maximum value of about 700 deg/sec, which is consistent with my previous finding [a]. The fact that a different strategy was employed by the birds in case of the flapping model bird suggests that motion plays an important role in distinguishing different levels of threats. At higher levels of threat, the birds react differently and also more immediately.

Modelling of Collision Avoidance Strategies: In this research [d], the collision avoidance behavior of birds was characterized by a set of guidance laws or algorithms. Here, I investigated how birds avoid mid-air collisions while traversing a stationary obstacle, using high-speed stereo video cameras to reconstruct their flight trajectories. My findings reveal that Budgerigars exploit some of the characteristic features of their cruising flight in the unobstructed tunnel, to avoid collisions with an obstacle when it is present in the tunnel. Analysis of these trajectories reveals that, when passing the obstacle from above, the birds often use a ‘flap-bounding’ mode of flight to scale the obstacle. Quantitative modeling indicates that the avoidance trajectory can be characterized accurately by an initial phase in which the bird applies a constant vertical acceleration to gain height, and a subsequent ballistic phase in which the bird closes its wings and passes over the projectile. The timing and strength of the acceleration phase are evidently tuned precisely to ensure that the maximum altitude is attained at the point of crossing the obstacle, to achieve a safe, injury-free clearance. These results suggest that the flap-bounding mode of flight, commonly used by many birds for energy-efficient cruising, is recruited by Budgerigars for an additional purpose, namely, obstacle avoidance.

The ballistic collision avoidance in [d] focuses on modeling of ‘flap-bound’ flight behavior of a single bird while avoiding artificial obstacles. However, we also investigated how two birds interreact while facing a head on collision. To study this problem, we collaborated with Robotics and Autonomous Systems Lab at QUT. Here, we investigated an inverse differential game approach to modelling the mid-air collision avoidance behaviors of birds. We proposed a general method for estimating the cost-functional parameters of a noncooperative differential game from partial-state measurements of an open-loop Nash equilibrium. We applied the method to model the behavior of birds while avoiding mid-air collisions. Our analysis suggests that a differential game model provides an accurate description of the observed collision-avoidance behavior and could provide new insights for the design of collision avoidance strategies for unmanned aircraft. This work [c] was published in the IFAC Symposium on System Identification, 2018.

a) Debajyoti Karmaker, Ingo Schiffner, and Mandyam V. Srinivasan. “Budgerigars adopt robust, but idiosyncratic flight paths” Nature Scientific Reports, 2020.

b) Debajyoti Karmaker, Ingo Schiffner, Julia Groening, and Mandyam V. Srinivasan. “Stretching Time and Space: How flying Budgerigars evade a moving obstacle” Submitted to IEEE Transactions on Intelligent Transportation Systems, 2022.

c) Timothy L. Molloy, Grace S. Garden, Tristan Perez, Ingo Schiffner, Debajyoti Karmaker, Mandyam V. Srinivasan. “An Inverse Differential Game Approach to Modelling Bird Mid-Air Collision Avoidance Behaviours” in Symposium on System Identification (IFAC), 2018.

d) Debajyoti Karmaker, and Mandyam V. Srinivasan. “Ballistic collision avoidance in Budgerigars” under submission, 2022.