Abstract Bibtex DOI Demo

Markers are the backbone of various cross-domain augmented reality (AR) applications available to the research community. However, the use of markers may limit anywhere augmentation. As smart sensors are being deployed across the large spectrum of consumer electronic (CE) products, it is becoming inevitable to rely upon natural gestures to render and interact with such CE products. It provides limitless options for augmented reality applications. This paper focuses on the use of the human palm as the natural target to render 3D virtual objects and interact with the virtual objects in a typical AR set-up. While printed markers are comparatively easier to detect for camera pose estimation, palm detection can be challenging as a replacement for physical markers. To mitigate this, we have used a two-stage palm detection model that helps to track multiple palms and the related key-points in real-time. The detected key-points help to calculate the camera pose before rendering the 3D objects. After successfully rendering the virtual objects, we use intuitive, one-handed (uni-manual) natural gestures to interact with them. A finite state machine (FSM) has been proposed to detect the change in gestures during interactions. We have validated the proposed interaction framework using a few well-known 3D virtual objects that are often used to demonstrate scientific concepts to students in various grades. Our framework has been found to perform better as compared to SOTA methods. Average precision of 96.5% (82.9% SSD+Mobilenet) and FPS of 58.27 (37.93 SSD+Mobilenet) have been achieved. Also, to widen the scope of the work, we have used a versatile gesture dataset and tested it with neural network-based models to detect gestures. The approach fits perfectly into the proposed AR pipeline at 46.83 FPS to work in real-time. This reveals that the proposed method has good potential to mitigate some of the challenges faced by the research community in the interactive AR space.

@article{dash2023interact,
  title={Interactions with 3D virtual objects in augmented reality using natural gestures},
  author={Dash, Ajaya Kumar and Balaji, Koniki Venkata  and Dogra, Debi Prosad and Kim, Byung-Gyu},
  journal={The Visual Computer},
  volume={-},
  pages={-},
  year={2023}
}
                    

Abstract Bibtex DOI

Various studies have suggested that computational thinking needs to be highlighted as one of the essential abilities and it should be taught in the standard K-10 curriculum. However, the standard K-10 curriculum does not always contain accessible technology that uses interactive teaching methods and efficient analytical skill development. Also, most practical computa- tional thinking applications that are being created seem to be complex and expensive for educational settings. In this work, we introduce PlutoAR, a marker-based augmented reality (AR) interpreter that is scalable, inexpensive, portable, and can be used as a platform for kids to enhance their skills in an inclusive way. PlutoAR incorporates AR through an interactive toolkit to give students the experiences of both the real and virtual worlds, overcoming the limitations of traditional and non-interactive setups. With the help of technology-based solu- tions, this effort intends to enable students from all demographic groups to engage in the learning process, regardless of their level of competence. The PlutoAR mobile application presently runs on any Android device with a camera. It creates inbuilt AR experiences like stories, basic elementary mathematics, navigating mazes utilising conditional loops to solve them, and the intuition of gravity. The usability of the PutoAR application is verified by performing the qualitative and quantitative analysis through user feedback. The application seems to be acceptable by users with a System Usability Scale (SUS) score of more than 80. In future, we intend to provide flexibility to the user for adding new content inside the application whenever needed and to promote AR-based collaboration among the students.

@article{dash2023plutoar,
  title={PlutoAR: a scalable marker-based augmented reality application for interactive and inclusive education},
  author={Dash, Ajaya Kumar and Behera, Santosh Kumar and Dogra, Debi Prosad},
  journal={Multimedia Tools and Applications},
  volume={-},
  pages={-},
  year={2023}
}
                    

Abstract Bibtex DOI

This paper presents a system for freehand interaction with 3D objects using gestures. A secondary contribution of this work is to present a system to recognize a set of suitable gestures which are used for manipulation of the 3D objects interactively, with the help of a deep learning framework using the 3D raw images captured via Leap motion interface. For this study, we have used our own dataset, a collection of images acquired using Leap motion controller comprises with six naturally occurring gestures. A deep learning framework built with CNN has been trained on this dataset and the validation accuracy has been found to be as high as 99%. This model is then used to predict the user's gesture to interact with the 3D objects with bare hands. The application has been tried and assessed by ten subjects. The subjects had no prior experience on how to interact with Leap motion sensor, making it an interesting study to explore the possibility of its usage by mass.

@inproceedings{tripathy2019natural, 
title={Natural Gestures to Interact with 3D Virtual Objects using Deep Learning Framework},
author={S. {Tripathy} and R. {Sahoo} and A. K. {Dash} and D. P. {Dogra}}, 
booktitle={2019 IEEE Region 10 Conference (TENCON)}, 
pages={1363--1368},
year={2019}
}
                    

Abstract Bibtex DOI

This paper focuses on using the augmented reality (AR) technology to create visual-aids through display for early childhood learning. The proposed methodology works on the principle of augmenting 3D virtual objects over the English alphabets that are used as printed markers. The important steps of a typical marker-based AR application are, (i) detection of markers in the field of view (FOV) of the camera, (ii) identification of the marker, (iii) estimating the pose of the marker, and (iv) rendering 3D virtual content over the marker in a live video stream. We have formulated the marker identification process as a classification problem which has been accomplished with the help of convolutional neural networks (CNN). The effectiveness of the marker identification process using CNN is validated by comparing its identification accuracy with support vector machine (SVM) classifier. The marker identification by the CNN model shows better accuracy than SVM. After successful marker identi- fication and pose estimation, virtual objects are rendered over the 2D projection of the alphabets. The seamless augmentation of the virtual objects over the markers are rendered on display. The setup has been tested on a large dataset and it is believed to create engaging experience for the kids, especially the kindergarten age group.

@article{dash2018designing,
  title={Designing of marker-based augmented reality learning environment for kids using convolutional neural network
  architecture},
  author={Dash, Ajaya Kumar and Behera, Santosh Kumar and Dogra, Debi Prosad and Roy, Partha Pratim},
  journal={Displays},
  volume={55},
  pages={46--54},
  year={2018}
}
                    

Abstract Bibtex DOI

Deep convolutional neural networks are becoming extremely popular in classification, especially when the inputs are non-sequential in nature. Though it seems unrealistic to adopt such networks as sequential classifiers, however, researchers have started to use them for applications that primarily deal with sequential data. It is possible, if the sequential data can be represented in the conventional way the inputs are provided in CNNs. Signature recognition is one of the important tasks for biometric applications. Signatures represent the signer’s identity. Air signatures can make traditional biometric systems more secure and robust than conventional pen-paper or stylus guided interfaces. In this paper, we propose a new set of geometrical features to represent 3D air signatures captured using Leap motion sensor. The features are then arranged such that they can be fed to a deep convolutional neural network architecture with application specific tuning of the model parameters. It has been observed that the proposed features in combination with the CNN architecture can act as a good sequential classifier when tested on a moderate size air signature dataset. Experimental results reveal that the proposed biometric system performs better as compared to the state-of-the-art geometrical features with average accuracy improvement of 4%.

@inproceedings{behera2018air,
  title={Air Signature Recognition Using Deep Convolutional Neural Network-Based Sequential Model},
  author={Behera, SK and Dash, AK and Dogra, DP and Roy, PP},
  booktitle={2018 24th International Conference on Pattern Recognition (ICPR)},
  pages={3525--3530},
  year={2018}
}
                    

Abstract Bibtex DOI

The regular K-10 curriculums often do not get the neces- sary of affordable technology involving interactive ways of teaching the prescribed curriculum with effective analyt- ical skill building. In this paper, we present “PlutoAR”, a paper-based augmented reality interpreter which is scal- able, affordable, portable and can be used as a platform for skill building for the kids. PlutoAR manages to overcome the conventional albeit non-interactive ways of teaching by incorporating augmented reality (AR) through an interactive toolkit to provide students the best of both worlds. Students cut out paper “tiles” and place these tiles one by one on a larger paper surface called “Launchpad” and use the Plu- toAR mobile application which runs on any Android device with a camera and uses augmented reality to output each step of the program like an interpreter. PlutoAR has inbuilt AR experiences like stories, maze solving using conditional loops, simple elementary mathematics and the intuition of gravity.

@article{singh2018plutoar,
  title={PlutoAR: An Inexpensive, Interactive And Portable Augmented Reality Based Interpreter For K-10 Curriculum},
  author={Singh, Shourya Pratap and Panda, Ankit Kumar and Panigrahi, Susobhit and Dash, Ajaya Kumar and Dogra, Debi 
  Prosad},
  journal={arXiv preprint arXiv:1809.00375},
  year={2018}
}
                    

Abstract Bibtex DOI

It is very difficult to count currency notes of different denominations in a bundle. We propose a system which can be implemented in paper currency counting machines to count Indian currency notes of different denominations efficiently and accurately. The system takes an image of standard orientation in upright position of Indian currency notes and detects its denomination accordingly. It makes use of image processing technique and artificial neural networks.

@inproceedings{nayak2015neural,
  title={Neural network approach for Indian currency recognition},
  author={Nayak, Jayant Kumar and Majhi, Chaitan and Srivastav, Apurva Kumar and Dash, Ajaya Kumar},
  booktitle={2015 Annual IEEE India Conference (INDICON)},
  pages={1--6},
  year={2015}
}
                    

Abstract Bibtex DOI

In this paper, a new method of image segmentation by histogram thresholding based on the concept of fuzzy measure minimization is suggested. The concept introduced here, uses extreme value type-1 distribution (Gumbel distribution) in order to define the membership function. The membership function is used to express the unique association between a pixel and its belonging region (the object or the background). The optimal threshold can be effectively determined by minimizing the measure of fuzziness of the image. The result of the proposed approach is compared with some existing methods and the efficacy can be verified over some standard images having various types of histogram.

@inproceedings{dash2015image,
  title={Image segmentation using fuzzy based histogram thresholding},
  author={Dash, Ajaya Kumar and Majhi, Banshidhar},
  booktitle={2015 IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (SPICES)},
  pages={1--5},
  year={2015},
}