Don’t let bugs get into production environments

“ If debugging is the process of removing software bugs, then programming must be the process of putting them in. ” – Edsger Dijkstra

I have written a number of blog posts on the quality of CRM projects/customizations/code.

• Why your CRM code and customizations should be simple
• Why rushed projects/code doesn’t save time and reduces quality
• The problems with complex code and complex CRM Customizations
• Bad code is like a virus, don’t get infected

During the process of writing these articles (and others) I have become convinced of the importance of keeping the quality of the CRM code high.  The quality of the CRM code is the foundations which the CRM project is built on.  If the quality of those foundations started to weaken the CRM project becomes harder to use, support, extend, debug.

CRM developers needed to keep standards high, quality of code high and stop as many bugs as possible from getting out of the Developer environment and into production.  This can be done by catching as many bugs in your Developer environments before they can escape to production.

Why is it so important?

There are a number of reasons why finding bugs in your development environment is better but main reason is time.  It takes longer to find and fix production bugs than development bugs as shown by this familiar graph

The picture comes this interesting blog post – Quality assurance in software development: When should you start the testing process?

The blog post highlights the cost of funding bugs in production

For instance, a bug identified during conception costs something around zero, but when this same bug is found only after implementation or test, the average cost of repair can get to something between 10 and 1000 times more than in the previous step. When customers find this bug in production environment, the cost of the problem considers all side effects related to it and. That is where things can get serious.

Below are my reasons to not lets bugs out of your development environment

Customers lose confidence when they find bugs

When customers find bugs, it makes them question the quality of the customizations being created and the testing process which has missed the bug.

Bugs frustrate the customer and the developer for different reasons but bugs can delay the testing/using of certain functionality, slowing the momentum of the project/set of changes.

Understanding of business logic fades but the code remains

When you are working on a piece of functionality you are in the flow, you understand the business logic, you understand the code, you are a developing machine.

In a few months time  your knowledge of the business logic will have faded, as will some of the knowledge of the code.  In many ways understanding how the code works is easier because you can follow the code but you might not remember what the code is trying to do or why (business logic)

• If you have to make a change to this code having unit tests will make it easier to test
• You can change the code and rerun the tests to make sure it’s still working.
• You can use the unit test to easily debug the code
• Tests show the developer what the code should do

Time

When a developer finds a bug whilst writing and testing the code, the developer can fix it quickly whilst they understand the business logic and code.  If the bug is found later the developer will need to reread the specs, ask business analyst and relearn the rest of the code so they can implement a fix.

If the bug makes it to production, the process takes a lot longer to fix

• A customer finds the bugs and raises
• The bug is logged by someone
• The developer will try and recreate the bug.  This could lead to more exchanges asking for more information about recreating the bug.
• The bug recreated and fixed (once the developer understands the code again)
• DEV testing
• Build and deploy in other environments and testing

Imagine how much longer a bug takes to be fixed when it is found in production compared to finding the bug in the Dev environment.  In the example above I have assumed the same developer is fixing the bug but often the bug will be fixed by a new developer which means the understanding of the code, fixing and testing all take considerably longer.

How do we stop bugs escaping to Production

In my experience activities like unit testing, integration testing and code reviews will help.  In my opinion the most effective way to stop bugs escaping to production is to improve the quality of your code.

Unit testing, Integration testing and code reviews will help find bugs but well designed code will reduce the number of bugs created.

Writing simple well designed code

The main method to stopping bugs is to make the code more simple.  Reducing complexity and designing your code to adhere to the S.O.L.I.D objection orientated design principles will create simple code.

Uncle’s Bob’s articles on SOLID

• The Single Responsibility Principle
• The Open Closed Principle
• The Liskov Substitution Principle
• The Interface Segregation Principle
• The Dependency Inversion Principle

Here are two more good summaries

• SOLID Object orientated design principles
• SOLID Principles of Class Design

Why is writing high quality code which adhere to OO design principles important?  Consider the core hiding place for bugs is within complex code and complex code exists where poor or no design of code exists.

I have written before about why code should be simple and the problems with complex code but I will recap.

Poorly designed code comes under many names, here are a few popular titles

• Spaghetti code
• Legacy code
• Big ball of mud

Poorly designed code hard to understand and is usually tightly coupled which leads to code changes affecting large parts of the code.  A change to one class/method can ripple through a large part of the system.

More dependences = more code which might not work after changing

Making changes to poor quality code is difficult, the initial difficulty is understanding the code.  Poorly designed code often has many dependences which increases the difficulty and risk of changing the code due to the increased risk of breaking other parts of the system.  Low quality code doesn’t manage the effects of change well

Badly designed code has no code reuse, the developers have to write new code or copy and paste some code (duplication, very bad).  Creating more code will take longer which is bad but the increase in code will increase the complexity, increase the lines of code to manage, increase the change of more bugs.

When working with legacy code (which is poorly designed code people have given up refactoring)

If the code is working leave it alone

Otherwise make as few changes as possible because it might stop working

There is a risk of unintentional bugs when changing legacy/badly designed code due to the code being complex and difficult to understand.  The cause of unintentional bugs is often responsibilities/dependencies which can link code/classes together.

If you have to work with legacy code or poorly designed code, these two books will help

• Working Effectively with Legacy Code (Robert C. Martin)
• Refactoring: Improving the Design of Existing Code (Object Technology Series)

Unit Testing

The main benefits of unit testing code is not finding bugs.  Unit testing will find some bugs but the real benefits come in improved design of code which is needed to make your code testable.  The biggest benefit I have having unit tests for my code is I can refactor the code with confidence because I can change the code and test it still works.  It’s easier to refactor and redesign the code to improve the design of the code.

Code is never finished, it’s never perfect, it’s exists in it’s current state,  ready to be improved.  I have written before Unit tests are a vital part of emerging code design.  Developers should be constantly trying to improve the quality of their code, simplifying, making the code easier to understand, reducing complexity, separation concerns, increasing cohesiveness of the code.

Developers with unit tests for their code see ways their code can be improved and refactor

Developers without unit tests think, if it ain’t broke, don’t fix it

Code Reviews

I think code reviews are a great net to catch bugs.  In the same way a writer finds it difficult to edit his own writing, developers find logic bugs in their own code.

Developers get so close to their code, it’s difficult to step back and impartially inspect the code, this is where a code review gets someone to not only evaluate the logic of the code.  Code reviews can ensure the quality and design of the code is high and can be an invaluable tool to learn and improve your coding.

Here some benefits of Code review

• Training/mentor
• finding logical bugs (many eyes)
• Increased code reuse
• Better use of core dll’s
• A second opinion

Here is a good post on Benefits of code reviews

Integration Testing

Developers don’t like testing their code, they prefer to spend their time creating code (and bugs).  Integration testing is a vital part of software development, to ensure the product is testing from an end users perspective.

CRM Software is complex and it’s difficult for one developer to understand and predict the various combinations.  Developers won’t use the software in the same way as the end user, integration testing will test the software the way a user will use it.

Conclusion

Productions bugs in my personal experience are a painful experience, usually involve plenty of time to understand and recreate the bug in your development environment.    The cost of fixing and moving the fix through the various environments is a cost in time, not to mention its boring.

The emphasis should be on quality and keeping the quality of the code base high by refactoring and improving the design of the code.  Most of a developer/programmers lift is spent inspecting and understanding code, which highlights the importance of keeping it simple and easy to understand.