One of my current roommates works at a government center for disease control research here at Currey Road. He is a biotechnology graduate and his work mostly requires him to collect data in experiments according to a strictly defined protocol. The word “protocol” here is key.

The experiments have to be performed according to a pre-established principle. Protocols become important in a setting where there is high chance of a minor mistake invalidating the entire set of results which draw on the data generated by these experiments. These biotech protocols usually require years to master as they need one to work with and control unstable chemicals, living tissue, dirty vessels, one’s own body, the environment itself. Dirt under your dirty fingernails can invalidate six months of your work in a second. My roommate over the course of the year has worked on just two of these experiments.

Lately, I have come to take an interest in such matters because of a development in my own work. My current project requires me to work with the Touch API in browsers: filtering unrequired palm touches when one writes with a stylus on an HTML5 canvas on touch devices in the natural manner one writes with on a sheet of paper. My goto device on which I have been testing and for which I am polishing my results is an iPad 10.

A solution to the problem lies with machine learning. Inspired by this paper I set out to train a decision tree in the matters of palm and stylus. How one trains a decision tree is by generating a relational (tabular) dataset columned with various characteristics of a touch contact (when I say touch contact, what I mean is a single touch gesture right from the start upon first contact, to the movements and until the end when you pick up your finger, stylus, palm or whatever you’ve set to rest on that screen) like the distance of the start contact from its neighboring start-points within a given time window, mean-radius and the like. Now, each entry in this dataset is labeled as palm or stylus and handed over to a decision-tree generation algorithm like the C4.5 or CART to juggle with. I take this decision tree, encode it to if-else clauses and use it in my final product or test scripts. How I collect my training data is by taking an HTML5 canvas, generating a 10px circle randomly somewhere on it and asking a lab-rat to try initiating a stroke from inside the circle using a stylus while naturally resting her palm on the iPad like she would on a sheet of paper. Strokes starting inside the circle are classified as “stylus”, the outside ones as “palm”. Simple right?

Except it’s not if YOU’RE AN UNDISCIPLINED AND INCONSISTENT DUMBFUCK. You see I am that way and this is an experimental protocol I am working with. I have the feeling and I think I am mostly right that when it comes to modern statistical machine learning, things are not quite different from my roommate’s stem-cell experiments. Machine learning is not the magical device of highly-paid wizards who sway their wand at any problem and the problem goes away, no Sir. Generating the training set with a well-defined, consistent protocol, transforming it, feeding it to the algorithm with perfectly tuned parameters (among the million variations possible thereof), and testing the results for statistical significance, mastering this entire chain of protocols is where a million bucks in salaries lies at. It always helps to be a mathematician of-course.

Let me tell you what I’m ranting about and where I screwed up:

1. I collected multiple data sets for the various kind of gestures like panning, zooming and others which did not fall within the category of my prime target (i.e. stylus or palm) and which needed to be positively filtered. I mixed them with my stylus/palm data set to train the decision tree. Except I didn’t label these data-sets as descriptively as I should have and ended up with hodgepodge, took bad guesses, mixed incompatible data-sets and ended up with unwanted decision trees. Do not ask me how many times I repeated this same mistake.

2. My test scripts compute their own characteristics on the raw touch event data that comes in. These characteristics need to be strictly computed in the same way the characteristics were computed during training. Turns out I initially wasted a lot of time due to bugs in my training code, testing code and inconsistencies in my own thinking which led to different ways some attributes/characteristics were computed during training and while testing.

3. Mismanaging the zoo of various decision-trees I generated by tuning the parameters in many ways and mixing them up while testing.

Now combine all these problems at the same time and there is an explosion in the lab.

Thinking of the big picture, there are questions which come to mind now that I have been working with statistics and in an experimental framework: how can one definitely tell if the inferences one makes from a dataset are true? Is testing the hypothesis on a different dataset the only way? Is a proof of the hypothesis possible in any way? Turns out this is the problem of induction as stated in David Hume’s A Treatise of Human Nature.

See you tomorrow,
Avi.

P.S. MORAL OF THE STORY: It never helps to be not disciplined. Always be disciplined. Chao.