Leaving IBM, joining Cloudera

Starting this August I'll join Cloudera as a Software Engineer in the HDFS team. Waving good-bye to my 4.5 year tenure as a Researcher, first at Oak Ridge National Lab and then at IBM Watson Lab. At this point I'd like to log my thoughts behind this choice. Around every job change there are two factors: compensation and happiness. The second factor, which includes sense of accomplishment, recognition from other people, and pure interest, will be the covered in this post.

Politics etc.
I didn't actually get particularly bad big company politics at IBM. Most of my Research or non-Research colleagues are very reasonable. However, in a company with 400,000 employees, it is very easy to step on other people's toes. You'll find yourself spending more time than you want on planning rather than doing the real work.

Another consequence of the large size is that executives know relatively little about the technical details, including the state-of-the-art. Meanwhile your performance score depends on how they like your demo. Therefore, the incentive mechanism is sometimes misaligned with what you know as the right things to do.

Breadth vs. Depth
During my 3.5 years at IBM Research I have worked on VM provisioning, cloud BFT, VM caching, software license management, and a little bit of HDFS as a side project. From my observation that's more or less the case for many researchers here. IBM's business model decides that we do a lot of "integration of XXX with YYY", and "XXX-as-a-service" here. I totally see the value-add and the technical challenges of those projects. However, I just prefer to go "crazy deep" on the XXX problem itself before moving to the next subject.

Things I will miss about IBM Research:

1. The "never-jam" Taconic Parkway. Seriously, I will miss Westchester -- easy commute to NYC, (relatively) cheap housing, Bear Mountain, ...
2. The "real scientists" running around the lab in white coats. Seriously, I will miss the resourcefulness -- you can grab an expert in any area (you name it, material science, partial differential equations, ...) if your project needs. 
3. The close connections with academia.
4. The T. J. Watson Center. The remarkable building, the beautiful library, and neat offices.

Things I won't miss about IBM Research:

1. "Welcome to the teleconference service, please enter your access code" ... "There are   17 participants on the call, you are joining as a participant". Seriously, there are just so many hour-long meetings where you only speak for 5 seconds.
2. Having to pay for coffee, having to use 4 year old laptop, having to try so hard to get a 23'' monitor. The HR department just doesn't consider it a priority to enhance employee morale and productivity.

Things I look forward to about Cloudera:

1. The "hacker mentality" and pride as a hardcore engineer.
2. A zoomed-in view of how people are using large distributed storage systems in production.