Microsoft Inking | Group of 2 | Contribution: Activity based Study, Protocol Design, Gesture Design, Interactive Prototype


Pen-Based-Gestures for augmenting digital writing experience and capabilities

Although the existing inking and note taking apps for tablet like Evernote, OneNote etc. are powerful, but they do not harness all the capabilities of digital medium. Features in these app are limited to most common functions like cut, copy, paste, undo, redo, stroke color changes, etc. The digital medium is very powerful and can do many more things like perform calculations, visualize, search etc. We are trying to embed these features in a note taking app without complicating its interface. We propose a novel stylus based gesture and keyword recognition system that can be used to access these features in simpler way.


Market is flooded with apps that use Inking capabilities, so it was important for us to understand- What all is already existing? What are most common features in inking applications? Gaps in the existing market? Simultaneously, a study of research papers helped us in defining some problems that users face while using inking applications and their solutions.

Design an app for touch devices that uses Inking (Pen based input on digital canvas) as the primary way of input and creation.

Understanding behavior of users

Users were questioned to understand their existing inking patterns and needs. Questions for user study encompassed all three aspects of inking i.e. notes, diagrams and drawings. Ten users were interviewed in the process.

We conducted as a way to observe notetaking behavior of users. We asked users to take lecture notes on paper and on tablet. We observed what interactions were natural on paper and how were they different as compared to tablet. Combining these we were able to observe some interaction patterns for the users. Two different activities were undertaken by 5 users each.

Design interactions for a digital note taking application which leverages the experience of paper and the power of digital medium to create more productive content for college students.


We came across some really interesting insights. Users performed micro-interactions like making side notes, “underlining” important points and marking content for later review using different symbols, markers etc. They used other tools in association with the activity of notetaking like calculators, convertors, graphs, etc. They didn't want to think too much while taking notes. They used non-preferred hand for interacting with paper: using one hand to move (reposition) the paper and other to write on paper.

Use of other tools
Use of other tools
Use of Non-preferred hand
Use of Non-preferred hand


Then we thought about ideas to solve the problem along with scenarios in with scenarios in which they could help. Two scenarios that stood out to us were using a second surface were we can keep notes and second using gestures to perform tasks such as adding reminders, highlighting etc.

Concept and Prototype

Augment capability of notetaking app by providing advanced features using gestures or keywords.

How many additional features does evernote provide as compared to paper? The current app like evernote, one note etc. have only few features like cut, copy, paste, undo redo stroke colour and width change, etc. But digital medium is very powerful and can do many things like perform calculations, visualize, search etc. We are trying to provide these features using a note taking app. These functions have common umbrella of accessing them i.e. pen based gestures/ keywords. A prototype with animation and gestures was designed using powerpoint.


Novelty of the app lies in the way user interacts with the system to perform common tasks. The gestures are adopted from the daily behavior of user while taking notes.