We would fail to write bubble sort on a whiteboard… Should we be proud?

Recently there was very interesting activity on the Twitter where famous programmers just admitted they are very bad at Computer Science (CS) and other fundamental concepts and still doing very well.

Now, this is very interesting. To me, this is just another manifestation, that knowledge has a little value today. Everything can be easily Googled in a very short time. Yes, the search takes some time, but it is much more efficient than trying to learn and remember all the details of CS and technology concepts. And I also admit: I would fail to write bubble sort on a whiteboard and estimate my algorithms’ complexity while being a successful developer for more than a decade now.

But one may come to the (I believe wrong) conclusion that it is ok to be unable to estimate the algorithm’s complexity and not to understand the essence of fundamental algorithms and their complexities. Algorithms are the methods for solving problems of procedural nature while design patterns are the methods to solve software structural problems and I would argue both are the very basics of software engineering. If you do not know these methods, when you are tackling the tough problem, you basically don’t know what you don’t know. You are unable to discover that your problem can be reduced to one of those fundamental problems and there is already an elegant solution to it. I definitely see my ignorance in CS as a problem and I am currently deliberately studying algorithms :-). And also clean code, which I always enjoy learning and practicing.

Here is what John Sonmez experience was after having learned algorithms:

All of a sudden it was like I put on special glasses that let me see the world in a different light and all these problems, all these places where I was like there’s nowhere in the real world where I’m going to use algorithms in my code, bam, it was popping up everywhere. It’s like, “Whoa! Look! Oh, I recognize this. This is like a min-max problem.” Bam! All of these places I started writing really efficient, really good code because I could see the problems.

I can also see the pain when job interviewers start asking puzzles and algorithms stuff. I hate it. If you want a job you are stressed already and now you have to focus, concentrate and tackle the tough problem at the blackboard as if you would do it at your real work! Problem-solving is very creative activity and It can not be done effectively under stress because the brain actually focuses on security aspect first and foremost. I don’t know how to replace this practice, but it is total evil to me. Interviewers, please, don’t do it to us, developers, unless the job requires this type of skill and knowledge. Pretty interesting article on the related topic from Yegor Bugayenko. But since interviewers still continue asking, maybe realizing the importance of fundamentals, we should be prepared!

Hence, if you have some spare time and you have a choice either to learn some new cool JavaScript framework, which will be forgotten in few years or some of the algorithms or design patterns which will serve you throughout all your career, I believe it is better to select last option.


Few guidelines on unit testing derived from my experience

When I first started using unit testing in my practice, I had no idea what I was doing. I was also frustrated. It made little sense to me to have everything unit tested because writing tests and preparing test data took me too much time.

Ever felt the same way? Don’t understand where and when to write test and where you should leave the code alone? Don’t worry, this is normal. This happens to everyone who tries to improve the way he/she does software. I can’t give you a concrete prescription, no one can, but in this post, I will share what worked for me in terms of unit/automated testing. This will, probably, work for you.

Don’t write unit tests for simple cases. One objective way to measure the return on investment (ROI) from unit tests is to measure how much time they save for development team by catching regressions. In simple cases, when the code is not going to be changed or code is pretty straightforward it is unlikely that you will get regressions, therefore it is likely that you will have no ROI at all from your unit tests. Furthermore, you will need to maintain them. The law of diminishing returns works here. You can get 80% of the benefit from covering only 20% of your code with tests. This 20% of code contain your core business logic, which delivers the most value to your customers. Everything else is some kind of glue code, configurations, mappings, frameworks, libraries interoperations and so on. The more effort you put to cover this code, the less ROI you will have.

Do large acceptance tests for refactoring. If you plan to do some large refactoring or restructuring, classical unit tests will not help. In fact, they will go into your way. Classical unit tests are small and test some small parts of the system. When you start changing things, unit tests start glowing red and you will need to delete them. The large acceptance test captures the whole business case of interaction between your system and the user. Such a business case is something that brings real value to the business and should not be changed during refactoring. Relying on acceptance tests will increase your chances to refactor without damage to the business. In developeronfire podcast, Nick Gauthier (episode 149) reported that he had his largest success in the career by moving the web application he worked on from classical client-server architecture, where HTML was rendered on the server, to the single page application (SPA). Acceptance tests made transition really smooth for his team. My refactoring team at SimCorp also had a big success of not jeopardizing our product’s quality by major refactoring we did. That refactoring touched almost every user screen in the system. My team lead insisted on having large acceptance tests suite, which eventually ensured our success.

Unit test complex algorithmic stuff by classical unit tests. As you probably know there is classical and London school of TDD. According to the classical school, unit test just applies input data to the system under test, harvest the results and compare them to expected results. According to the London school, unit test invokes system under test and then checks if it behaves as expected, i.e. it calls its collaborators correctly and in the correct order. While I feel frustration from unit testing simple cases, I get a lot of value from classical TDD when I develop complex algorithms. Value, in this case, comes from regressions which happen during initial development, because when you develop a complex piece of software you can potentially spend days in one unit trying to put things together. I vividly remember one programming exercise from my SimCorp days. I had to develop a program, which would take APL data structure of any complexity (for instance matrix of matrices) and generate APL code which would recreate this particular data structure. My first attempt failed because after 4 ours of work I was far from being done. And most of this time was spent on retesting program operation with different types of inputs after every change to the algorithm. Next day, I have tried classical TDD. And the process was not only fun but also fruitful. In about 4 hours I was done with approximately 30 tests. I remember my impression was that I would not have finished such a program in this amount of time and confidence without unit tests.

Apply London school TDD to the integration code. What if your core business logic is not algorithmic one? What if you bring the value on the table by integrating different systems together? For a very long time, it was an open question for me. In such a cases I still want to be sure my core code is tested well. However, classical tests are awkward because integrators often don’t even have inputs or outputs. I believe London school tests are perfect for them. Once, at StrategicVision, I had to develop a system that would download videos from video hosting services, extract audio from those videos, transcribe audios and finally save transcriptions and video links to the database. No business logic in a classical sense, right? My code just invoked video hosting web service, then invoked downloader, then invoked transcription web service and finally invoked repository to store results. I wrote a bunch of tests which were testing, for instance, such facts: if the system under test invoked downloader for a particular video, it should later invoke clean up for this video; if the system under test invoked database repository to store results, before that it should invoke transcription web service.

These guidelines are highly subjective of course, but at least they work for me, at least at this point in my career. Hopefully, you will also find them helpful.


Refactoring disappointments

During my university days, I was a big fan of refactoring. I believed it should be done as much as possible in every situation when you encounter a code smell. But university days meant a lot of books and no real practice. Migration to the industry, specifically to SimCorp taught me that my beliefs were not true. Refactoring can be harmful in certain situations and it also can be really disappointing and demotivating activity. Here are some cases which I experienced in my career:

Refactor-distractor. Refactoring definitely takes away your time which could have been used for development of new features or fixing bugs. Now, sometimes refactoring is worth doing when you can clearly see the benefit of it when you can get 80% of improvement with 20% of the efforts. For example, you can see that few hours of refactoring can help putting important functionality under test or improve understandability of the frequently changed code. But often you will find yourself doing code improvements on the part of the system which rarely gets changed or doing improvements to the code which is already good enough. Diminishing returns law works here. You are pushing towards better code but into every further code improvement, you need to put more and more resources. I don’t know about you, but when I got caught by this type of refactoring, at the end the day I always feel pissed. The feature I am expected to deliver is not progressed and resulted code is not ideal because it is just never ideal.

Refactoring with no domain knowledge. Avoid doing refactoring if you don’t understand the domain of the software very well. This lesson was delivered to me by SimCorp. They have extremely complex financial software which they produce for a very long time. Every system which is in production for a long time has a lot of legacy code, SimCorp Dimension is no exception. And here I am, a freshman developer who just defended Ph.D. on refactoring and given the task to fix some bug deeply in the core of business logic. I discovered a function which length was over 7 hundred lines. It was definitely bleeding for refactoring and I immediately jumped in and starting extracting new functions. But there was a daunting problem! I did not know how to name those new functions. I had the best intentions and good technical knowledge but these prerequisites were far from enough to be able to do worth refactoring. I was running around distracting my colleagues and asking what they thought was the best name for this or that piece of code. I have finally done with this refactoring. It took few days and I was already exhausted and disappointed that my bug fix took that much time. But this was only the beginning of the trouble. Now, when something went wrong with this code, people started redirecting bug reports to me, although I absolutely was not aware what this code did in terms of business logic. No one actually came to me and said thanks for this clearing up! And this is because they were not happy with my changes at all. They knew the code well and now I have changed it and gave awkward names to the functions. Without domain knowledge, I made things even worth. Even after almost 4 years of my work there my, now good friends and colleagues, joked about names I gave to extracted functions.

Large refactoring by a dedicated team. As I said I was huge refactoring fan when I started at SimCorp. Therefore I joined the refactoring team which was forming soon after I joined this great company. The task of the team was to improve very badly rotted part of the system. Although I learned some good lessons from this experience, in about a year I understood it was a bad idea to have dedicated refactoring team. My discovery was supported by plenty of blog posts and books which described similar situations and came up with the similar conclusions. To make a long story short – it is very hard to justify for a business an existence of a dedicated team which does not deliver a tangible value. We completed only one project in a year which was only the part of big refactoring, although we worked pretty hard. Personally, I realized I wanted to deliver value to customers, it turned out it was a motivating aspect for me. Just re-coding what was already working well turned out to be a little fun to me. So I left the team and very soon it was dropped by the management. Most developers agreed that refactoring should be done gradually as a part of features development.