AskErnest.com

Decoding Video With JavaScript

Ernest Wilkerson — Wed, 05 May 2010 12:36:34 GMT

With all the hoopla surrounding which web browsers will support HTML5 video (and which codecs they will support), a buddy of mine mused about decoding video in JavaScript. I mean with a canvas tag, people have been able to produce some amazing things. And these browser JavaScript engines just keep getting faster and faster. So isn't it remotely possible that some level of video decoding (of certain video formats) in JavaScript would be usable? And it's pretty cool when one begins to think of the different possibilities. I admit that most of the possibilities I'm thinking of might be used for some sort of pseudo-DRM. But I could also imagine some pretty nifty uses for it too. There are, if I remember correctly, some sites out there that allow HTTP download of torrents. What if there was a way to preview a video before they it was completely downloaded? What if the first X pieces of a file were downloaded and available for preview before the torrent was complete? That way, a user would be able to determine if they were retrieving what they really wanted.

I could even imagine subscription videos. For instance, using a JavaScript-based decoder in conjunction with chunk/byte-range XHR requests, a server/service could limit you to the amount video you're allowed to watch. For instance, you would pay from X GB or MB of video time and the server just rings that up to your account every time you watch a chunk. So you only pay for what you actually watch -- not the whole video. And service could also implement some sort of pseudo-DRM. With enough talent, one could create one's own encoding scheme and use rotating time keys or something for "DRM". Of course, you hand over your decoding code when the user hits your site - but minification of JavaScript has to be the most (un)intentionally effective obfuscation I've ever seen.

But anyways, I'm not saying these things don't exist or are not already possible -- just that a completely ubiquitous approach to it is becoming more and more feasible. The only big piece missing from the pie, so to speak, is a way to write to an audio buffer. And the developers of FireFox are working on that. Hopefully, their API becomes standard much the way Apple's canvas has become a standard (except for IE). As a proof of concept, I've implemented a pseudo-H261 decoder in JavaScript. It's not entirely...uhmmm...accurate...but it works for what I wanted to demonstrate. It can decode an intra frame. No motion compensation routines etc. were implemented for inter frames/macroblocks -- it just decodes one picture. When I figure out (from the spec) how motion compensation should work -- I'll have it decode multiple frames. On my laptop it can decode a CIF intra frame in about 25 ms and render it in under 100ms -- in Chrome. FireFox is a little slower -- around 68ms and 120ms respectively. The times are a little slow for full motion video (I think), but this doesn't account for multiple opportunities for optimization. There are many inefficiencies in my code that could drop the decoding and (especially) render time -- probably by a significant margin.

(note: I've only tested in Chrome and FireFox. IE will not work. Safari is an unknown but it should work.)

You can check out the decoder here. You can download the files here.

Microsoft Research presents...Infer.NET

Ernest Wilkerson — Thu, 22 Jan 2009 03:28:47 GMT

Happened to be cruising the Microsoft Research projects page...looking for something cool to mess with. Lo and behold, I came across Infer.NET. I could tell you what it's about...but it's better to get it straight from the horse's mouth (so to speak).

PDC 2008...Here I Come

Ernest Wilkerson — Wed, 02 Jul 2008 16:12:46 GMT

Just finished registering for PDC this year. It'll be my first time attending. I'm extremely excited (like a kid in a candy store). Now only if I could think of a good idea for the "ShowOff"...I'd be set. Everything I can think of is just a mash-up of stuff that's already been done. *Sigh*...oh well.

Face Detection in (pure) C#

Ernest Wilkerson — Sun, 04 May 2008 00:57:31 GMT

Have you ever heard of the Open Source Computer Vision Library (OpenCV)? Well if you haven't, you might want to head over to the project's Wiki page or Intel's project page. In short, OpenCV is an cross platform open source library (written in C++) that provides a great deal aid for developing computer vision applications. OpenCV can be used for biometrics, security, human-computer interfacing, etc. The brilliant people behind OpenCV have given us the ability to quickly implement such advanced techniques as Face Detection and Fast Object Tracking.
About a week ago, a friend of mine told me a he was interested in implementing some basic face/feature detection features in a web-based surveillance system (well...a viewer at least). Now, after some research, you can quickly come across some .Net wrappers such as OpenCVDotNet, but what he needed had to be either Java or 100% C# (so he could convert it to Java...haha) so that the functionality was as (easily) portable as possible. I chose to write a quick proof-of-concept in C# (at least for the face detection piece). So I did my best to quickly port the face detection functionality from OpenCV. OpenCV is great, but there are some knocks against its face detection...the most prominent ones being that it doesn't take lighting into account enough (it won't find faces in some images that other libraries will and it will report false negatives sometimes) and the fact that it may actually violate a patent.
Below you will find what my code as a library project, and a sample project that uses the Yale Face Database. Keep in mind that the code below is just a proof-of-concept and that it can definitely be optimized. I've also skipped doing image histogram equalization which might find more faces in certain images. OpenCV also uses a k-nearest neighbor algorithm which i didn't really bother to port...I just used Euclidean distance (I'm not sure if that is what OpenCV was already using)...which should work most of the time. In my testing with the Yale Face Database, faces can be detected in ~1.4 seconds on a 300x300 image. That's nowhere near realtime detection, but that's mostly due (I think) to the bounds checking in .NET arrays and lists. Quick profiling show that for a 300x300 image, there are about 52 million array accesses in that ~1.4 seconds which ain't too bad for C#. So here are some scissors....run with 'em.

Face Detection Library
Face Detection Sample Application

Office Fun: Key Flipper

Ernest Wilkerson — Tue, 19 Feb 2008 03:07:54 GMT

I read in my bathroom reader that the top three "problem" employees according to the Wall Street Journal are: "The Non-Stop Talker" "The Screamer" and the "The Practical Joker." I guess that makes me a problem employee then. I don't scream as much I as used to, but I do play practical jokes when appropriate (hehe). Recently we had one of our programmers give his two weeks notice, and because he's a good natured guy (and he's quitting) -- I thought it would be appropriate to give him hell this week. Seeing as I'm a developer, my pranks tend to leverage my aptitude/proclivity for system intrusion and wreaking random havoc. So today (Monday) I decided that I would make a program that hooks his keyboard input and types a predefined, yet configurable, amount of text -- no matter what keys he presses (ex. type "0123456789" and the output is "I love ham"). Hence -- the KeyFlipper program which leverages the KeyboardHooker library. The KeyboardHooker library exposes a KeyboardListener class that can be used to log keystrokes or stop them. The library also exposes the KeyInterceptor class which does all the "real" work required to torture said co-worker(s). This program also goes in the largely useless bin...but seeing the look of joy on his precious little face more than makes up for it. Tomorrow (Tuesday) will feature something similar involving hooking the mouse. Ahhh....I can't wait until tomorrow.

KeyFlipper (requires KeyboardHooker)
KeyboardHooker

Convert Web Pages to PDF

Ernest Wilkerson — Wed, 23 Jan 2008 03:02:41 GMT

Today I decided to write a little program to capture every "screen" in our web-based application for regression/documentation purposes. We're undergoing major changes and I wanted something static to use as a reference. Granted, we have now (and will have in the future) digital copies of the current (soon to be previous) version of our application, I feel alot better about being able to just open up a notebook and look at what the application used to look like.
So, the bulk of my work consisted of capturing the graphical output of the Microsoft WebBrowser ActiveX control. I was initially using the .NET control, but experienced massive issues trying to cast to MSHTML interfaces that I knew the objects must inherit from. Yay! But anyways...Capturing input from the ActiveX control involved p/invoking good ol' GDI32 and blitting the output from the WebBrowser control onto a bitmap of my creation. Once that was done, I just took my fancy new bitmap and stuck it into a Adobe PDF using PDFSharp (which is quite a handy library).
There are multitudes of articles on the internet about capturing a web page to an image, but none of them seem to deal with the issue of pages that have dynamic content (frames). My solution addresses this issue and also has a little bit of error handling -- just in case you happen upon an error page. By default, I have that functionality disabled though because the scanning is done in a very inefficient manner.

Well anyways...here are the solution files if anyone ever needs 'em.

New Host...New Year...New Stuff

Ernest Wilkerson — Sun, 20 Jan 2008 17:53:46 GMT

So a new year rolls around, and I look forward to being a year older. Not that it really means that much...the year changing just helps to further distort my perception of reality. As a child of the 80s and 90s, I still think of the late 1990s as being "recent", even though it was a decade ago *sigh*. That aside, it's nigh-on-impossible for 2008 to be as life changing as 2007 for me. I begin this new year in a new state, as a married man, and (more importantly) as a father. I remember writing a similar post to this one on my WordPress blog almost exactly a year ago.
At that time, I had no idea how much my life would be changing. Well, I had some inkling...I knew I was getting married, but that was about it. Nothing can surpass the joy that comes from being/becoming a father. The one thing I really look forward to this year is watching my son grow up. His birth has definitely been the highest point in my short life -- and every day since has run a close second. His birth sparked me re-registering this domain -- though I waited until now to actually post anything. Up until now, I was just using the domain to host pictures so my family/friends could readily retrieve them. Unfortunately, I don't have the motivation to migrate all my older blog posts to this site...oh well. Yeah...new host...new dad...2008 here we come!