Categories
Blog: My thoughts

Quit, Quite, Quiet : A Year has gone

It is interesting how these three word summarize my last year. I quit lot of things that were stopping me. I quit to go down. However it is not entirely true that I succeed in my quest. Though, I am certainly at better perspective of life.

I have quite interesting events happen all around me, and most recently I start my own company which looks to expand all around and always. I was surprise that how small change in motivation leads to totally different perspective. My client feels more closure to me, and are getting happy with improved service of mine. My client always like my work, but somewhere with time, I lose the touch where I deliver to the desire. My biggest issue was time period. Though it is surprise me that when I start to reduce my work hour, I start to get more done and client are more happy as the delays are going away. Now, to get more done, my company is expanding its work base, and it means I have lot more to focus on. I am quite prepared for it and doing things as they come.

I remain quiet for long time on blog, as I wasn’t sure what I should write. I was not learning much of new tech and there is little that I do differently which is not already explain in other blogs. Though I still had few topics to write, technical or personal. But, they do not inspire me to write a post. Today, with this post I try to break that barrier so I can be more focused on things that make me, what I am. Sharing knowledge and learning from other is one of my biggest strength. I will try to write more and learn more. I start enjoying programming again, this new generation of programming was going over my head before, but now I found myself in the zone. I can learn and overcome them. Hope to post some technical articles soon.

Categories
Blog: My thoughts Operating System PHP Server Configuration

Linux Upgrade from 12.04 to 14.04: Save the day

So, this fine sunday, I think I wake with “lets spoil the day” mood. I start by checking my server to see how it is doing. A regular log view and updates I perform on my VPS. all was good, until I realise an update is available for my billing software. I decide to purchase an update for it. Before buying I forgot to check system requirement. Well it is just an update, and we install it every here and there for other things. So, I have updated copy of my billing software, which I follow the steps of upgrade and upload to my live server. All went well until I reach for software to suggest install PHP 5.6 instead of version I had. Well no problem how hard it is to update PHP in ubuntu 12.04, just matter of running apt-get.

Soon, I realise PHP 5.6 was never release for OS, and I need to use PPA. Found an old tutorial teaching me how to use PPA as it is my first time for it, the blog explain everything good, found couple more article but they are nearly the same. So, I went on and install the PPA and update the PHP 5.6. I run a command line to see what version of PHP is there and it correctly shows 5.6. So far so good, but suddenly “If anything goes wrong will go wrong” statement hit me. It only update CLI PHP not apache version. Damn, what is wrong with it.

PPA on Ubuntu 12.04 LTS need Apache 2.4 and Apache 2.4 never makes to Ubuntu 12.04. So, again there is PPA for apache but that doesn’t work for me well. I mean it does update few things, but cannot make PHP’s Lib for Apache to get installed. At this very moment, I have two options. Either I rollback all updates to my Backup Copy, or I move ahead to upgrade OS to 14.04 LTS. I choose later one.

Simply run “do-release-upgrade” and follow the course of stream. but it stuck as VestaCP’s APT repo didn’t work with upgrade. Not sure why, I just remove them from APT repo’ List and all goes well. All updated nothing is broken but Apache. It is broken because my previous attempt of PPA apache upgrade leave it broken and in conflicting state.

I remove and try reinstall apache without PPA, but that cause more conflict. After hit and trial with PPA and non PPA apache, I end up getting it upgrade and everything running.

There is no technical details shared in this post, as this post is not for Step on how to do it, but what not to do it. And that is only one thing, never miss reading about what upgrades you are going to do, never miss system requirement before paying someone. It only took me 4 hrs to get it all going, and frankly it turns out to be Positive for me, as it get my pending upgrade of OS done. But, I would have save my day if I read the simple requirement before buying the upgrade for billing software. I would have avoided unnecessary upgrade path, and reduce the down time to say 1 hr at most instead of 4 hrs. Good thing my blog is not a business, but never try that at client server.

Categories
Article Blog: My thoughts

Next step to development team

In my last post I try to define what motivate me to learn new things. Also, I try to show that I really believe in learning new things. In this blog, I will try to define the basic things a team need for developer productivity. This list may not be what most agree, some might see it incomplete, some might found it over burden. But this works well for me till now, and if I found something better I will change this. But from my experience in sub-standard teams and project this will be a good start.

First, there needs to be a team. Yes, that is first requirement because often there is no team to work, there are individual who just do their work. The team consist of at least two entity, at least two human. Now, you might question that I work alone so how I am referring a team to be of two entity. So to answer it, the first person in team is the Client, and you are the second person, the developer. Client is always the first person in team. He/she must be ready to accept you as his team. This is where team builds. If, he just think he hire you for the job, then he will only get a contracted human resource. He will never get a person who even try to understand his business needs. And if that happens the game is over before it starts. This doesn’t mean the role of developer team is reduced, or all the burden is on client to make team. Developer should be truthful. He should be the one, a client can trust on. It doesn’t matter if you fail to achieve deadline, if doesn’t matter if you fail to deliver the expected product. What matter is at end client doesn’t think he is been cheated. He doesn’t feel you are not skilful enough for job. He should not regret his decision about choosing you.

Second, part of ideal setup comes, with tools you use. How you protect what is been done? How you ensure reliability of software/code you are writing? How you ensure that it works on load? What will happen, in case you are not available to maintain the software? Are you using out date technology? Are you using immature technology? And most importantly, how does it affect client and his requirement? I often see these question are neglected at small scale. Developer mainly think, it is small project and not worth taking all extra in account. If we reduce those Version control, documentation it cost less to client and we gain project over competitor as we offer same solution at low price. If you are looking for developer, ask him to explain his stand on above question. And if you like answer then only choose him. I am not saying everything must always be included, but if you understand the risk/profit of not include all these it is win situation.

Third thing is communication medium. This is another important setup. Often, developers are not available to receive phone calls or chat. They reply email after 2-3 days, while client wonders if you are still alive. Same happen when developer ask for feedback and they get a reply after a week saying, “yes”. Whereas developer was asking to choose between two possible scenario or asking to define a problem. You must set a communication channel between all members of team. And there must be a definite time frame between replies. Even if there is nothing to reply, just acknowledge that message is been received.

Once you master these three things, a first step in ideal setup is been done. If you are developer, please always be truthful to client. I am tired of doing Editing job of some other developer’s failure. I want to work on new project more. I am not your debugger.

Categories
Blog: My thoughts

Learning is GOOD

I am working as developer for quite sometime now. I worked as developer in couple of companies, then hire few to assist me. But never need or know the concept of unit test. Or rather I would say it was forgotten to small team like us. But as I start working more and more as one man army, I start to realize that I need something that will help me test what I develop. I need something that make my code readable to me if not world. I need something that make my life easier.

This was never a thought for me, or most developer around us. But as soon as I have to work on old projects that my team start developing and left in between, I realize that I am missing something that make me a better programmer. But, today after practicing few good concepts like unit testing, clean code, code documentation, code formatting, life has been much simpler. I know to any good experienced developer this sounds like, “What? You are talking basic here”. But reality is most 1-200 people companies I worked in around here do not follow these stuffs. They mostly care only to deliver some visual to client so they get paid. This has been a bitter truth around, that never impress me.

I knew, this is not right and what I am doing, no matter how good I do, I am doing it wrong way. So, I start learning new concepts, start upgrading myself. Now, I can see myself standing near the edge of developer crowd that deserve to be called a developer. I am not part of that crowd which just struggle to survive because their code always fails.

If you are starting your career, it is important to find a better company even if he is alone like me, than a companies that pay you better salary, but lack the way a developer should work. In my future post I try to define what needs to be learned for a beginner. As I want beginners to not repeat my mistakes. I don’t want to waste their life’s 10 yrs figuring it out that they start at wrong place.

I am looking for some developer to assist me, if they are ready to work like professional should. But I am very unfortunate, as I only meet countable few which are ready to take on better steps, most only sounds like they want to do good. My hunger for learn will never die. I can comfortably says, learning is my food for life. It is new things that keep me on edge of my seat, and I will learn everything I can.

I am no more a Desktop Only Programmer, I am not only a Web developer, but I am a Mobile App Developer and I can built app for Apple, Android and Windows. This is addition to my skills of Windows Desktop, Linux command line tools, web apps and Web site add-on.

I just keep repeating to myself, that I should always keep looking something to learn. I will keep doing that.

Categories
Blog: My thoughts MS SQL Server Programming SQL SQlite

Reading Large Binary Files: Child Play

Yes, that was funny title, but after your experience it you will agree to me. So, here is the story. I have been working on a software that read some recording from hardware device to database, we have 45 records per second for 30 days, so it is about 30x24x60x60 record entry with 45 columns in it. We have to make a desktop application so we choose .NET for it. The first version of software was released by my company 3 yrs ago, and the reading well, we were inexperience at that time to manage that data, and what we get is about 2 hrs to read all that data to database. Oh, I forgot to tell that 30 days entry was from one hardware device and we have 3-4 device :), so we took 2 hr to read all say 4 devices. Now that is not acceptable thing. So, we decide to rewrite the complete software to make use of some parallelism, as that is only way my team thing it is going to work.

I start the rewrite, and with only hope to reduce to 2 hr work to 30-45 minute I start writing code, but this time we make a exception from last time, instead of using TEXT ascii file or SQLite database, we opt to use SQL Server to store our data. Reason, well first pre-release version of software use Text file, we never get that part working for more than 15 days, and it always get out of memory for one or other reason. Then we start using Sqlite which is 5 times lighter on hardware and speed the reading and information access, however, there is part which still use text file. So, in order to avoid two source we opt for database only, and Since client already have Sql Server on seperate machine, we thought it is good to have seperate machine storing database, for long term and obviously LAN benefits. Since client already have SQL Server and we are using .NET I decide to go with Sql Server, only.

So, we start reading binary file, and start putting insert query for each record [just for testing the lowest speed], and it goes through in 12-13 minutes. wow, we already reduce 30-40 minute job to 12 minutes, just by using full time database. Now, the next challenge is to speed it up with known bulk import methods. couple of them that I tried are

1. Using Dataaset and Update feature of Dataset,

2. Using Long Insert query to send 100 or 1000 records in one “Execute”

3. Using SqlBulkCopy feature in .NET. This is obvious choice on speed, but in few cases it fails for me, so I have to look for first two options as well.

So, at end we get SqlBulkCopy as our tool to go with, now it doesn’t simply import the data in Database. We have to prepare background for it, SqlBulkCopy is used to send data from CSV files, so we create CSV file from binary read, than import this file to Staging Sql Table, and then transfer data from Staging Table to main table, all this is done in 2-3 minute flat. Yes, a 40 minute work is done in 3 minutes. Period.

The trick is, we reduce the number of operation needed to write to disk, we still create CSV, but perviously we are creating XML File, secondly, we have multiple file write procedure going, we remove all this. Infact to achieve that speed we even stop writing LOG file to trace error, we use SQL server to record those error for us. Infact removing Log Text file with SQL error log, itself speed things from 5 minutes to 3 minutes. though logs ar enot that long, but disk writing is very slow as compare to database insert.

All said, I will hardly use plain text file in future for long data read, it will be some sort of database for sure.

Categories
Blog: My thoughts

StackOverflow.com: A Review

I lately start using StackOverFlow.com a lot to get my answers and to help others. It is really one good site, one can hook around for a while. I love reading some of their questions. Some of those questions are that I got around in future as developer myself. The fun thing about this site than all other past forum and such site I use is the reputation points and badge. I love getting those, though I am still not very active on site and hence my profile is not very good yet. 

However the thing I don’t like it negative on questions, well actually I love it but most of users are not understanding the concept well on either side. I have seen questions like “Here is my SQL query” and then a query, but what about it? marking such question negative is good idea, but then I saw a question “My SQL query doesn’t produce result, here is my query …” that question too get negative, because person forgot to tell the error he got, but from query it was clear a syntax error. Well that is how some people read and reply.

Couple of my question has answer where from answer I can clearly tell that person even didn’t read my question properly. They simply try to paste a piece of code they have used or found, without understanding the fact that the question is not about coding but concept of piece. 

Another thing, I observe in my short time on site, is that few questions seeks a answer to syntax error, but are actually conceptually wrong, and funny thing is it got 4-5 answers all telling him different way of writing the piece of code involved, but none understand the need of person [which is obviously not written in question]. 

Seeing such situation, frankly I think I can easily suggest my fellow programmer that a person who is asking the question is either not clear on topic or not in very good state of mind [might be frustrated of his failure on topic already], but person who is replying is replying for sake of helping him, so if you want to help such person try to understand what he want to achieve and what he want to do. rather than giving him negative or useless answer. What is the point if your shared knowledge is actually a burden on community. It is a responsible job to answer in public community, “Workaround” is the last thing to tell. In programming is you are not analyzing the question before you answer actually tells how easy it is for you to lose concentration on job, and software you are making is not really going to be very effective. 

I love the site and love the concept, I used lot of forum, but will stick around on it for quite a while. However for that I need to get more time from my job. But for a new programmer, it is my understanding that if you read through the forum it will increase your knowledge. 

Categories
Blog: My thoughts

Azure: First Experience

After a long time finally get time and reason to touch base with Azure Service. It was great experience working with it thus far. My client want to move his website from a regular host to something that we can easily manage. We are not keen to hire a server admin to do stuff for us in detail. Anyways, Azure give us platform just for that. 

So, I try to create VM on Azure as when we talk to Customer representative about our requirement they suggest us to use VM instead of website. And it was super simple to create VM. Well, i use other service to create VM or reconfigure VM, and Azure ask little more questions, but still if you know what you are doing it is not hard to answer those. 

Once we have VM up and running we use SQL Azure to install 5GB database and configure it by importing database using SQL Azure Database Import Wizard a community managed Wizard software. Things really went smooth. 

What impress me most is the fact that 1 GB of backup file gets downloaded in just 4 minute straight, I have use lot of big name VM host, but really most of them do not have that much speed as I see here. In fact some of dedicated server that I use do not have such high internet access speed.

The interface is fast, I really going to test it further to see its extreme. As my client says, I work on it as if I am using it 10th time. I never feel like I am doing it for first time. 

Anyone want his site to move to Azure.. I am really looking to do that for you.