If you are a customer of TD Bank this past week must have been hell for you. A few months ago TD has aquired Commerce and last weekend they decided to switch over all of the Commerce users to the TD platform. It was a disaster. Direct deposits weren't posting, checks weren't clearings, money couldn't be withdrawn, there were late and NSF fees charged to many TD customers.
My wife and I have a legacy Commerce account and this experience was a wake up call. Commerce was great and we loved it but ever since TD Bank came into the picture the pleasant experience has been destroyed. This debacle was the final straw for us as their customers.
For me as a software engineer, however, there are some interesting points here that I would like to go over. When I worked for JPMorgan Chase I learned that Chase is made up of hundreds of smaller banks that they've aquired over the years. I don't recall reading or hearing about Chase's customers ever experiencing a disaster like this. So, why is it that some banks can go through a merger smoothly and TD Bank couldn't ?
First, let's consider how many banks operate nowadays. Customers use ATM machines, checks, tellers, online banking, debit cards, etc. all throughout the day. Some transactions show up immediately as "pending", others don't show up till next day. Over night the bank runs a batch process that consolidates the ledger files from all of these different systems and business units, reconciliates, and updates customer's transactions. Pending transactions become completed, scheduled bill payments occur, etc.
So, my first question is why do banks still need batch jobs ? In this day and age of instant messaging systems and nearly continuous connectivity, why can't transactions be posted in real time ? I suppose there may be cases where the connectivity is down and customers may freak out that they can't withdraw, but there are ways to ensure fault tolerance and failover. Worst case customers can be told to go to a branch in another town.
Imagine if TD's brokerage unit Ameritrade was 24-48 hours behind at posting equity trades. Chaos would ensue! So, obviously TD has access to real-time transactional technology that the rest of the bank can utilize.
Second, let's consider the basics of a software lifecycle. Most of us in the field would agree you don't just dump a new release onto a couple of million customers over night. According to TD, what happened was they migrated Commerce people over to the TD system and voila the overnight batch process runs twice as long. That was a predictable factor and I don't understand why the processing capacity wasn't increased prior to the release. Did anyone perform load testing ? Why wasn't the roll out done gradually, a handful of customers at a time ?
These are the technical questions I'd like to think about as an engineer. And, as a customer, my wife and I are going to be closing the account at TD over the next few weeks as we unwind the checks and move to a new bank.
