Australia and Computer Vision

Note: I have more than ever been busy with work and the loss of reach my blog suffered when Google Buzz ended has reduced a bit of the motivation I had earlier when updating it. I won't say I'm making a resolution for a come back this year but I will try to write earlier and write often.

The Three Sisters, a rock formation in the Blue Mountains in New South Wales, Australia.

I was recently lucky enough to attended a big conference in computer vision, the ICCV - International Conference in Computer Vision. It took place in Australia last December and it was refreshing. Australia is a great place to visit but more on that later. One thing that took me by surprise was that I and my collaborators were awarded the prestigious Marr Prize in computer vision. I'm very grateful about that. I won't go into details about our own work that led to this prize in this blog post but I will borrow an excerpt from Tomasz's vision blog to describe it:

"[...]. It is all about entry-level categories - the labels people will use to name an object - which were originally defined and studied by psychologists in the 1980s. In the ICCV paper, the authors study entry-level categories at a large scale and learn the first models for predicting entry-level categories for images. The authors learn mappings between concepts predicted by existing visual recognition systems and entry-level concepts that could be useful for improving human-focused applications such as natural language image description or retrieval. NOTE: If you haven't read Eleanor Rosch's seminal 1978 paper, The Principles of Categorization, do yourself a favor: grab a tall coffee, read it and prepare to be rocked. [...]"

I not only recommend following his blog but also to check out and fund his new venture on Kickstarter.

The conference was a big success and we saw some of the new trends in the field that are getting more established. I will briefly discuss two of those trends: 1) Using geometry for high level computer vision tasks like detection or total scene understanding. 2) Using neural networks and particularly deep architectures. These two separate trends share some history. Geometry was initially taken as the main tool for solving vision problems but time seemed to prove that, at least for object recognition, representing images with 2D templates and using clever machine learning was sufficient to recognize objects. This approach has practically solved the problem of detecting faces. Geometry was to an extent confined to a separate track of classic tasks like reconstruction from multiple images. But in this new trend there is a clear revival for geometry in recognition where people have realized that in order to move forward to make recognition more general we need to incorporate geometry back into the mix.

In the same light neural networks were at some point more or less thought as a general black-box for learning. You show the computer some images and some labels and the neural network learns how to label new images by adjusting weights and filters on the input representations. This weight and filter system has some analogy to how networks of neurons work together, hence the name. They were also used together with 2D templates to learn to detect things like digits or faces and they were successful. Unfortunately there was the problem of scaling to more general neural network architectures with more layers. This required adjusting an enormous amount of parameters that computers were just not able to handle at the time. Machine learning theorists also came up with very solid theory for other alternatives to neural networks like max-margin classifiers and kernels that had better theoretical grounds and comparable performance than the neural networks of the time. And soon people largely stopped using neural networks to attempt computer vision problems and even problems in other fields. This recent revival and interest in neural networks in computer vision is due to recent key successes in training deep neural network architectures that have shown superior performance on several important vision tasks. Today's computers combined with large amounts of data and clever new techniques have all made this possible.

Our paper on entry-level categories doesn't fall in any of the above two categories but instead insists on yet a new focus on vision which is that of finding clever representations of what we should be learning about our visual world. Solving computer vision is not just about translating pixel content into labels but rather about building a higher representation model of the visual world that we can eventually apply to specific instances of this visual world. We have advocated for some time that we can augment this visual world model using knowledge collected from text and studying how people describe images using natural language.

I will take the almost impossible task of using my last paragraph to highlight Australia as a destination. It is an awesome place! Being a huge island, the fauna of Australia is just so different than everywhere else that it feels the closest to a totally different planet. I can only imagine how was the first encounter of non-aborigins of the island with those animals. The kangaroos are like a portrayal of deers taken from a very creative sci-fi movie except they are real. The koalas also have no parallels. The wombats and tasmanian devils are also something to be set apart from the rest of the fauna. I also had a chance to visit Cairns and the The Great Reef Barrier and while this is an awesome view of the underwater fauna I have to say that I enjoyed the above ground fauna the most.

Visual Attention and Visual Saliency

"Everybody knows what attention is ..."
-William James 1890

This quote is referenced in a research paper from the Visual Computing group at Microsoft Research Asia (MSRA), titled "Learning to Detect a Salient Object". I don't know exactly the context for that quote but it is interesting that somebody says this in 1890 when yet today we don't know many things about visual attention. Visual attention is particularly interesting in Computer Vision because in this field we want to teach computers how to recognize things in the visual world and it seems humans might be taking advantage of things like visual attention in ways computers still aren't.

To actually give some definition of visual attention I would say that it is the condition by which our vision focus in more or less degree on some things within the total amount of information that is perceived. Particularly in computer vision there are some research lines that are closely related to ideas in visual attention, one of them is visual saliency, which could cover among other things a) class-independent object detection or proto-object detection (although proto-objects as defined in the visual perception literature might not be directly usable in a practical application), b) detecting salient objects on an image (under the assumption that we humans do not consider all objects are equally visually important) or c) Detecting saliency maps that define regions that are important on an image without explicitly associating them with an object.

The paper from MSRA ("Learning to Detect a Salient Object" CVPR'07) detects a salient object under the assumption that we know a priori that there exists a salient object in the image. I believe this assumption holds for a large number of images on the web because that's just the way we think when we capture pictures, we usually focus on something. It is easy to imagine that not only Microsoft but also Google are already using some form of visual saliency to autocrop images from the web for display on search results or generating thumbnails. But beyond this obvious application there is room for using these kind of techniques to improve object detection itself or at least to avoid trying to detect objects on every possible location within an image.

Sample saliency maps for the top left image used as features in  the MSRA paper.
Those maps were generated using my own implementation of their method.

The MSRA paper also introduces the MSRA Salient Object Database, a large collection of images with manually annotated bounding boxes enclosing the salient object on each image. The only thing not included is source code, that's why in 2009 while I was starting graduate school I decided to implement their method on Matlab [link to source code]. And although the CRF formulation is not exactly the same, I get similar performance to the one reported in the original paper (See slides included at the end of this post). The paper has got some considerable attention since it was first published and so although I don't keep track of how many people are downloading my code, I see a lot of traffic coming from Google search. Also I didn't run many experiments beyond what is explained in the original paper but I found somebody using my code who did a more thorough evaluation. This was done by a student at the Computer Vision class at the University of Texas Austin http://vision.cs.utexas.edu/cv-fall2011/slides/larry-expt.pdf. As I had expected this method does better than Itti & Koch (previous much simpler approach) but only when it actually detects something, which is most likely to happen in the kind of images where we have a clear single salient object, the kind of images the method was trained on.

Links in this post:
MSRA: Learning to detect a salient object source code:
http://www.cs.stonybrook.edu/~vordonezroma/code.html
Saliency Experiments Slides from the University of Texas Austin:
http://vision.cs.utexas.edu/cv-fall2011/slides/larry-expt.pdf

Colorado Springs and Kinect

I have again neglected my blog for sometime due to deadlines and other aside work. I just realized I never got myself some time to write about my visit to Colorado Springs for CVPR 2011 during the summer. Actually there is not much for me to say about Colorado Springs since I hardly visited any places beyond the hotel so I will just mainly refer about one particular paper presented in this conference.

CVPR 2011 might perhaps be remembered as the conference where the research paper Real-time Human Pose Recognition in Parts from Single Depth Images (The Kinect paper!) by Microsoft Research was presented. (Ok, this is obviously an overstatement, quality of research at this conference is really high) Microsoft Kinect is a product that has had a big impact that goes beyond gaming. This is a very iconic example of Computer Vision that works and is readily available to the world.

Wearing a cap and being Kinect captured

Kinect has inspired hackers (http://kinecthacks.net/), artists (http://artandcode.com/3d/) and general technology enthusiasts since its introduction some time ago. It has also inspired researchers to create new algorithms that can clean the data captured by the Kinect sensors and make the most out of it or just play with it (here http://acberg.com/kinect/ some kinect hacking in Matlab by Alex Berg, one of my vision professors in Stony Brook). The picture I included in this blog post was ironically captured in our fancy Motion Capture Lab using the inexpensive capturing device from Microsoft. People here have also been working on Kinect with applications to Music performances and Motion identification. (More links to be added later...). Update: Interactive Music using Kinect: http://tamaraberg.com/papers/kinect_music.pdf

Although I didn't stay for the whole week of the conference I also presented a paper in CVPR 2011 about automatically estimating photo-quality and user engagement for photographs titled: High Level Describable Attributes for Predicting Aesthetics and Interestingness. Our goal was to use Computer Vision to recognize what are the kind of photographs that users think are cool without explicitly having to ask them what is cool? (Note: Users might not even realize what are the individual things that make them judge something as cool or interesting). 

High School Algebra in Latin America

I'm grossly generalizing here in the title of this post, I will mostly discuss about my own experience as a high school student in Ecuador. But as I have read from several sources, some of these things indeed apply to lots of places in Latin America. That being said, I will keep the title.

If you come from one of the spanish-speaking countries in Latin America, you might with high probability recognize the book cover in this blog post. This is what used to be and is the synonym for algebra for most high school students in those regions. If you don't recognize this image, be it because you're not from Latin America or you really didn't know about it, then I would really like to know what's the standard book used in your country for learning algebra. This book is so pervasive in lots of places in Latin America that the word Algebra and Baldor and the picture of the guy with the turbant all come together to people's mind when the word 'algebra' is uttered.

What most people don't know is that Baldor was a Cuban mathematician who later emigrated to the United States and not the guy with the turbant depicted on the book cover. I couldn't find much information about his education in mathematics or other published material from him but he held a teaching position in the United States later in life, although the book was already being distributed from Mexico at the time. I believe one of the reasons for the adoption of his book was the lack of mathematics textbooks in Spanish at the time. But mainly because this is also a good book and if it has any flaws I believe those are the same flaws that books in other languages might have and I will refer to this in the following paragraphs.

One thing that I particularly like about this book is that it's well organized and easy to follow, I would say even easy to follow on your own. One thing I don't like is that most of the exercises are repetitive and easy, even sometimes boring. One thing that I like is that every few pages it has short biographies for every major mathematician in history. This is probably the first book where students get to know who are Laplace, Euler, Descartes, Newton, Fermat, etc. Guys who you will keep on hearing from, especially if you go into the hard sciences later in college.

Finally I will quote something said by the great physicist Richard Feynman in his interview with the BBC and which I think illustrates a criticism that applies to this book as well as to books in other languages regarding teaching science:

"I learnt algebra fortunately by not going to school and knowing the whole idea was to find out what x was and it didn't make any difference how you did it, there's no such thing as, you know, you do it by arithmetic, you do it by algebra, that was a false thing that they had invented in schools so that the children who have to study algebra can all pass it. They had invented a set of rules which if you followed them without thinking could produce the answer: subtract 7 from both sides, if you have a multiplier divide both sides by the multiplier and so on, and a series of steps by which you could get the answer if you didn't understand what you were trying to do."

I think this applies to this textbook and I highly doubt Prof. Feynman was ever in touch with our legendary Baldor's Algebra. Sadly, this is just the way Algebra is being taught in schools in general: learn the rules and get me the results.

Related Links, Sources:
Richard Feynman's interview
Aurelio Baldor's wikipedia entry

Update (April 30, 2011):
I just want to clarify that even though I mention this Baldor Algebra book was adopted maybe because it was a good book giving the standards of the time and available books in Spanish, I also state that it's too easy and boring and even if it was more difficult then it would still fall in the mistakes other books fall, just teaching calculations and rules and not the beauty of math. The idea is better pictured in the following video, this is a TED Talk by Conrad Wolfram on his view of teaching math.
http://www.ted.com/talks/conrad_wolfram_teaching_kids_real_math_with_computers.html
(Thanks @sergioroa from DFKI - Germany for sending the link)