Making the Most of What We Have

An interesting article on keeping a positive outlook.

Sometimes, seeing the silver lining is all you need to keep your business burning bright.

These are the few things the author highlighted:

• Stay positive because it helps you be creative during stressful times.
  


• If the seafood is going to spoil anyway, you might as well enjoy a great meal while you can.


• Keep a sense of humor at all times.
  


• Vote with your wallet.

I always keep these few values in check. And I will continue to make sure of this.

Looking back at my previous company, I was always there at the most critical moments in the company. There were major cases of database corruption, critical performance problem with the application, screw-up of processes. The only critical moment that I was not involved physically involved urgent migration of application & DB to another machine due to server hacking. That night, I was unable to sleep well, not because I was worried (everyone should know I sleep like pig) but I received calls multiple times throughout the nights. That night also, due to stress, the 3 people involved ( 2 operation guys and 1 developer) started to come out with funny names for the machine, database, folders, etc.. For e.g. we found names like "the-return-of-application-x", "appxOK" and the name of the 3 persons are found everywhere.

It maybe the most stressful times, but, unbelievably, I think they enjoy it as I also enjoyed all those crazy moments. I am not joking about either the criticalities or the enjoyment. They were all heck of the problems. If the problems were not solved within 1 day, there would be serious implications and I might say, our customers could sue us or take away the business. And that may be the end of the company.

So, let me recalled some of the pressuring yet fun nights I have had:

• In earlier day of deployment of a major application in the company (early because we only have few customers using the application back then), we stayed back really late. At one time, we stayed through the night and straight headed to eat dim-sum as breakfast at around 5 a.m. We spent long hours for deployment because we only have one machine, so we shut down the application, backup our database, deployed our changes, performed testing, and restored backup database before we started back the application. Believe me, all these are done in a production machine. When you do not have the resources, you will make do with whatever you have.
  
  Well, we did some mistakes back then because of this also, like during the testing, the application would sent out email alert. Well, at sometimes, we forgotten to reset the emails in our "test database" which caused the real customers to receive the testing email.
  
  
• When the same application grew, of course, we started to have other dedicated machines for testing so we abandoned our old RISKY practice. Well, to be honest, not a lot of people could do that, switching live application to test and vice versa. Only the most careful and talented bunch of people could do that, ahem, which included me. Obviously.
  
  Well, as the application grew, we started to have more programmers working on the application. That was when another problem arose.
  One day, the application started to have serious performance problem. It was down every 1 hour or so. We were busy the whole day, while trying to find the root cause and solutions, we needed to actually "pressed the button" (actually, is click on the mouse to restart the Window Service) to restart the application every now and then.
  
  That night, the 3 person responsible to find the solutions (2 developers (included me) and 1 operation guy) got home early. Wait, we were not lazy people. Neither were we irresponsible. We just got home, took bath, packed our stuffs, met each other somewhere and had a very nice dinner of mixed korean and japanese cuisine. Then we headed to the data center and spent the night there troubleshooting what was wrong. One of my colleague then also had bought some vitagens drink and instance cup noodles. So we were like eating the whole night because it was a little too cold at the data center.
  
  We did some testing on the application, ate, ran through health check on the machine , drank some vitagens, ate again (this time asked one of the guy to buy some food outside), ran some profiling tools to detect the problems, ate again, then looked into the log. After sometimes, it seemed like there were some connection leakages in the application.
  
  Only at around 7:30 a.m., we were only able to find the root cause of the problem. We were already giving up and were ready to call our boss to give him the bad news. That was when I suddenly remembered we have done a deployment the day before. And when I ran a check on the changed code, alas, found the method that opened the DB connection but did not close it. The changes were done by a programmer who just joined the company back then.
  
  It was actually a no-brainer to find out the root cause if only we can put all the bits and parcels together but only managed to find out after the whole night. But, maybe it was the cold or the constant eating that keep us from thinking logically.
  

• On another case, at around 10 p.m., I received a call from the operation guy that the live database was corrupted after he tried to extend the table space. Seeing the hours needed to spend in the office, I packed my clothes, my cuddly Winney-the-Pooh toy (need the moral support!), wore my spec and waited for the guy to come to my house to pick me up. He came and then we went to 7-11 the 24-hours shop to buy instant-cup-noodles , food and drinks like we were going to picnic. Well, to keep the story short, we tried to restore the database from tape, only to find that the restoration cannot be done. Thus, we dug up the database backup to disk 1 week ago, restored that and then started to import all the documents backed to the system to make it up-to-date. We were still doing this, when the people started arriving at work at the next day. I was still wearing my ugly spec. (I looked ugly and geeky in it). People also started wondering what had we been doing all night there. Then we had breakfast with our boss downstairs. Definitely, he was very thankful we made it (again) and because also we need to wait for sometimes to see whether the application was running fine . Only around slightly before noon, then we were able to go back. Well, I still can recall the feeling of collapsing to the bed like a dead person.
  
  
• As I tried to recall more nights spent in the office and I think there were at least 4 more. But since I have written so long and my mind is a bit fuzzy right now, I just briefly talked about them. Some of the nights involved running stress testing on the test machine because the application is having some performance problem, some documents were missing and need to be recovered, very slow DB and SQL (somehow found that some of the tables were analyzed automatically. Why this cause the slowness? If you want to know, buzz me personally and I tell you) . And the most recent of the 2 occasions before I left my ex-company were the database migration and the live database could not be started after being shut down for a short maintenance.

So, looking back, I felt I had been there. The worst possible situation, maybe. And I have gone through them without feeling really panicky and stressed out. One ex-colleague asked me this, why I did not seem to feel the panic and actually looked quite relax. I told her, what problems does it solves when we feel panicky and stressed out. It will not help the situation at all , make us unable to think well to solve the problems and also affect those around us. We should not be afraid of our mistakes or problems, but embrace it with positive outlook that we are going to solve the problems no matter what.

Well, recently, I have been through a lot of things that make it difficult to stay positive. Uncertainties loom ahead of me, different environment, different people I see everyday, different sleep patterns ( it kills me to wake up so early these days), deteriorating health (I have been having diarrhea for past 1 month - on and off), etc. But, when I think of those critical moments where I had to stay up for the whole night to solve those problems before the sunrise the next day, when I still can remain positive and calm while still having some fun, I guess I can still take on more things currently and still be positive.

Maybe, like what my friend suggested, I should try to blog about more relaxing things like relationship thingy. Who want to continue to hear me rambling about mindsets and my job challenges anymore? LOL

Sometimes, we do not need to be too hard on ourselves. Forgive our own mistakes but look for ways to improve and change for the better.