Thursday, August 04, 2005

Production Switch Over .....




We planned production system will shutdown at 7.00 AM. By 1.00 AM (next day) upgrade to new environment should be completed. Manufacturing team will test till 5.00 am. No issues by this time then we will release the upgraded system .If at all any issues we will revert back the system.
Production system is down at 7.15 AM. We started working. It is really challenging for our team to meet the time lines.
This is first project for me working at onsite while the production switch over is going on. For HCL this one is a first of its kind project. HCL before never handled this kind of projects.
The first activity to my module is to upgrade the OS. Next activity is installation of oracle client. In production server CD-ROM is started giving problems. We tried to ftp oracle client from development environment. But development system is a half-duplex system, so ftp is very slow. We changed the CD-ROM hardware and installed oracle server. Due to this problem we are 2 hours behind the schedule. After that everything went on smoothly. By 4.00 PM we completed all checks and our team declared that our modules are fine and ready to go.
After that I had done some front-end activities to upgrade database. My last activity started at 9.00 PM. Basically we need to install XML in four servers. In my testing environment I have written a script and it is running successfully. In Development we have cc compiler. But in the production servers we didn't have cc compiler and we have gcc compiler. When I run the script it showing compilation error. I told client PM we need a cc compiler. He told that cc is licensed software. We can't run trial version in production and asked me run with gcc compiler by changing the options. I changed the Makefiles and tried. But still problem is remained. I tried several options. Client PM told me previously DBA installed this software and check with him. Now he time is 10.30 PM. We need to release the hole system by 01.00 AM for production testing. I worked with DBA. He also got same errors like me. Now the time is already 1.00 am and it is released. Our Installation is not a showstopper for basic testing. Production people stated testing. But we should done at least by 3.00 AM. After that we need to test this one. According to our plan if any problem occurs by 5.00 PM which is potential show stopper in the new system we need to revert back the whole system. Then we tried with lower version of gcc compiler. It is able to compile some files. We run the applications. Thank god now it is responding. We installed all four servers by 2.30 PM. From manufacturing people there are no issues till 5.00 AM.
We released the system for production at 7.00 PM. Now it is in live. As of this time there are no major issues except some database synchronization issues. After 34 hours I return back to room.
I got fever so I have taken rest. Now I came back to office. Ever thing is going fine. Client is very happy about the performance of the system. Many mails are threading up to congratulate us. We got succeed. Personally I got some good quotes from my project manager. I learnt a lot form this project.