NYC Java Engineer

Process vs product

If I were asked what my greatest flaw is, something that I struggle most with, I would say the following: I care much more about the process than I do about the product.

There, I said it. I believe the road is more important than the destination. When applied to software engineering this means that having a solid process in place and following it without exception is more important than the schedule and the product. In the end, following the process results in a higher quality result.

I feel very strongly about this.

Project managers struggle with conflicting requirements. They must accommodate for both stakeholder demands and process integrity. In the face of stakeholder demands it is the process that is typically broken. Features are sneaked into the production, shortcuts are taken, and things otherwise proceed in a way that I would disagree with. Events like this and this serve as my vindication and reinforce my value system.

That puts me at odds with management and results in immense pressure and personal stress. I could easily agree that sneaking features into releases and scope creep is just fine, but that goes against my most deeply held values as an engineer. That said, I have never been a project that followed a process to the letter.

Whether you perceive this as a character flaw or a strength doesn't really matter. I know full well it is not possible to have a perfect project. I need to find a way to cope with this or otherwise I will never be happy.

"Hello World" using Apache Thrift

Apache Thrift is a remarkable piece of technology. It is orders of magnitude more light weight than any XML or JSON based protocol and it is much easier to use than SOAP, CORBA, or EJB. I wanted to cover some of my bases before I recommend it at work, so I wrote a hello world program.

First, you need to download and install thrift compiler. You can find all of the information you need on the thrift site. It takes awhile to build it from sources, so make sure you have a good cup of coffee ready.

I wanted to write a hello world program and benchmark it against a native Java implementation to see how much overhead I am incurring. This implementation looks like this:

public class HelloWorldNative
{
        public static String hello(String name)
        {
                return "Hello, "+name;
        }
        
        public static void main(String args[])
        {
                long start=System.currentTimeMillis();
                for (int i=0;i<100000;i++)
                {
                        System.out.println(hello("world"+i));
                }
                long end=System.currentTimeMillis();
                System.out.println((end-start)+" ms");
        }
}

On my laptop this completed in 1217 ms. Now lets try using RPC with thrift. First you need a hello.idl IDL file:

service HelloWorld {
        string hello(1: string name)
}

You compile it using thrift compiler like this:

thrift --gen java ./hello.idl

Thrift can generate client and server components for all of the languages it supports, including Java and C++.

Now you need to implement the handler for your server:

public class HelloWorldHandler implements HelloWorld.Iface
{

        @Override
        public String hello(String name) throws TException
        {
                return HelloWorldNative.hello(name);
        }

        public static void main(String args[])
                        throws Exception
        {
                TServerTransport transport=new TServerSocket(9090);
                TServer server=new TThreadPoolServer(new TThreadPoolServer.Args(transport)
                    .processor(new HelloWorld.Processor(new HelloWorldHandler())));
                System.out.println("Started");
                server.serve();
        }

}

HelloWorld class is generated by the thrift compiler and inside you will find interfaces and classes for you to use. Had I not already had a HelloWorldNative class I would've written mine from scratch implementing the HelloWorld.Iface interface. It's a good practice to keep your code in sync with DDL as closely as possible.

The main class starts a multithreaded server on port 9090.

Now you need the client code:

public class HelloWorldClient
{

        /**
         * @param args
         * @throws TException 
         */
        public static void main(String[] args) throws TException
        {
                TTransport transport;
                transport = new TSocket("localhost", 9090);
                transport.open();
                TProtocol protocol = new TBinaryProtocol(transport);
                
                HelloWorld.Client helloClient=new HelloWorld.Client(protocol);
                long start=System.currentTimeMillis();
                for (int i=0;i<100000;i++)
                {
                        System.out.println(helloClient.hello("world"+i));
                }
                long end=System.currentTimeMillis();
                System.out.println((end-start)+" ms");

        }

}

Run the client code. On my laptop it completed in 4885 ms. Given all the extra plumbing that is in between the client and the server I'd say this is not unexpected.

Some interesting things to explore are -- how do we pool connections ? How does it behave in a multi-threaded environment ? We know it all works because of our experience with Cassandra, but we wanted to use Thrift for our purposes how would we go about it ? But for now, this simple "hello world" works.

More on classifying software engineers

The other day I wrote a post about two kinds of developers . I am now not convinced that such a linear classification is good enough.

Consider the chart I sketched out above. It is best to classify software engineers in two dimensions. Along one dimension there are platform developers vs application developers. Along the other dimension are developers vs maintainers.

We have already discussed what platform development vs application development means. Now lets talk about development vs maintenance.

A developer can take new requirements or ideas, design a solution and code it. They may have to research new technologies, apply new techniques, and find and learn how to use an open source library. People with this personality would otherwise be tinkering with an open source project or learning a new framework if they were unemployed. To draw a motorcycle maintenance analogy, these are the people that design and build a new motorcycle.

A maintainer can take an existing application or platform and enhance and fix it. Maintenance is not separated from development. It requires a different attitude towards work, however. A maintainer must learn how to build upon an existing code base rather than invent a new one from scratch. This is the person who would rather be fixing an existing motorcycle rather than designing a new one.

It is highly possible that someone with a developer personality will not be happy in a maintainer role and vice versa. Maintainers are results oriented, more interested in the completed application rather than in the project itself. Developers are project oriented, more interested in the project rather than the result. A maintainer may remain on a single project for years. A developer will get bored and unhappy within a year unless rotated to a different project.

Developers should not be hired to do maintenance tasks. Maintainers should not be hired to do new development tasks. A mismatch of personality types can be detrimental to the project. It is up to management and team leads to figure this out and properly identify software engineer personalities and skill set.

Two kinds of developers

There are two kinds of developers out there. There are platform developers and there are application developers. Both are equally important in the software engineering ecosystem but they have fundamentally different personalities and approaches to work.

A platform developer has the creativity and ingenuity to build software frameworks and to put seemingly disjointed building blocks together. He can build a software platform that business applications can be built upon. He can identify patterns and opportunities for code reuse. If you judge a platform developer by their ability to deliver to customer's requirements they are never going to succeed.

An application developer has little or no interest in building a platform. He has a grasp of customer's requirements and he knows how to use the platform to build a business application. His interests lie in meeting immediate customer requirements. He has the creativity and interpersonal skills to build great applications. If you put an application developer in a role where they have to build a platform they are going to be confused and helpless.

These two roles are not necessarily incompatible with one another and they often work on the same teams or in close collaboration. A good example of this is salesforce.com. There are platform developers behind the salesforce cloud platform. There are application developers that build great applications on top of it.

Neither category is worse than the other and both are important. The trick for a team lead or an IT manager is to identify these two different groups early on and assign people to the right roles.

How Oracle is destroying Java

When Oracle acquired Sun I was not happy about it. To me at meant sure decline of Java. This proves me right.

Arduino kit at RadioShack

Very excited to see Adruino kits for sale at RadioShack. Reminds me of DIY Sinclair kits in the 1980s.

Food for thought on software processes

I was at a Big Data networking event sponsored by Datastax and had an interesting conversation.

If you are a large fund, say a pension fund, and you are doing due diligence on your brokerage and you want to know whether they have proper software quality assurance processes in place there is a very easy way to get an answer. Find out whether your own IT department has proper software engineering processes in place. If your answer is no then you are not doing your best for your customers that you expect done for you.

Financial firms trade and compete with one another and chances are no one really manages software in a safe and predictable fashion.

Treat your customers same way you would want to be treated. Manage your IT systems in a way that you want your service providers to manage theirs.

Ubuntu Phone OS

What the world needs is a Yet Another Mobile OS.

Have computers become too complicated for teaching ?

I learned computer programming on a Cold War era Soviet programmable calculator called Elektronika MK-61. It was a very simple device that used a four element calculation stack, a handful of registers, and programming it was very much like writing assembler code. It had a number of undocumented features that made simple games possible. It's cousin MK-52 was used as an on board computer on a Soyuz spacecraft.

The point, however, is that it was a ridiculously simple device. There was no user interface to write home about. There were no objects, no persistence, no class hierarchies. To teach kids how to program was a matter of discussing sequences of mathematical calculations, writing them out to use postfix notation instead of infix, figuring out a way to only use a stack of four numbers deep, and then writing a program. If the kid is ready to solve a system of linear polynomial equations, she is ready to program a calculator to do the same.

My first computer was a Sinclair ZX Spectrum. It had a built in BASIC interpreter that also acted as a rudimentary operating system. To load a game you had to know at least one or two BASIC commands. Writing a simple platform game was remarkably easy to do.

This was in the 1980s Soviet Union. Fast forward to now. Before a student can even write a program they have to go through the process of learning how to use computers which over the years have become remarkably complex machines. Sure we have nice user interfaces but they are far from simple. They require a rudimentary understanding of how to use a mouse, a keyboard, how to download and start an application. The built in tools for programming are far too complex and far too specialized. Shell scripting on the Mac and Linux, batch files on Windows - neither is conducive to an environment where a kid can write their own version of PacMan or Tetris. Environments like Xcode and Eclipse require a college degree to even grasp what they do, while languages like Java are too advanced to teach at the middle school level.

So, how do we introduce programming to children ? Algorithmic thinking is an important skill for a 21st century world even if you don't end up becoming a software engineer. I have shown my 6 year old daughter how to program in MIT Scratch. To spice things up I put the Scratch itself on a USB stick and showed her how to load and save her programs. She seems to get it.

What is needed, however, is a very simple computer that boots into the BASIC interpreter much like the home computers of 1980s. Programmable calculators fulfill this goal to an extent and by all means should be introduced in schools at a very early stage. But nothing excites the imagination as a more tangible computer with tools that help a child produce a shareable executable program they can show off. Raspberry Pi is extremely intriguing and I am tempted to order one. But then - my kids are still too young to appreciate it and I am too busy, but I know a day is coming when I am going to show them how to get a small inexpensive computer do amazing things.

Linux to drop 80386 support

Engadget reported a few weeks ago that Linux is going to drop support for 27 years old Intel 80386. Thank you, Engadget, for reminding me I am getting older.