fbpx
Tag

Developers

October 16, 2018

Is software getting worse?

My question targets not only desktop software but also mobile apps, web applications and websites. Seems like software is getting worse and it won't be better. And I just have to start my rumble with the worst - mobile apps.

A very good example is Tinder. This thing needs almost 25 seconds to load and takes 50MB of space. The usability of profile page is very bad (oh, you don't need to sort your photos), application sometimes don't want to load profile pictures and transitions between pages are lagy even when the application is fully loaded in the memory. And we are talking about something, what is basically a chat (not even a good one).
You would imagine that with such high earnings, company would be able to easily craft better user experience and improve the performance. But why, when new phones with better hardware are released constantly? We are getting better hardware, but sadly the user experience is still same because the software is getting slower. After few Android updates, I can install only few applications because the internal storage can barely fit updated OS. Even though my phone is far superior than fist version of Samsung Galaxy S, the loading times are somehow slower.

Desktop software isn't better. When I tried to install Visual Studio last time, it took forever to download and install several gigabytes of data. The size of whole package can be for sure explained, but it still surprises me how can a software package take so much space. We aren't talking here about computer game, where executable is several times smaller than resources like textures, sounds, music and 3D models. It also contrasts against exceptions, which often comes from open source territory. These exceptions can offer minimal footprint and rich features packed in single executable file. Visual Studio also install several gigabytes of itself on system drive, even though you selected different hard drive. Not nice, when your system drive is SSD with limited space.
Another concerning trend is rising usage of web technologies in desktop applications. When I saw an Atom text editor for the first time, I was excited to try lightweight alternative to bloated (don't mean it in a bad way) IDEs for coding. The editor is indeed faster comparing to full fledged IDE, but it's hardly lightweight and smooth user experience. When you compare it against C++ based Sublime Text, the difference is noticeable. I'm scared of the day when web technologies will take over the desktop environment. We are slowly getting there. Sooner or later, we will start seeing advertisements in the software we daily use. Heck, Windows 10 is already doing it.

Someone can see a poor performance as a non-issue. You could even say that most desktop apps are pretty fast. But let's take a look at video games. They have to do a lot of things in one second - handle tens or thousands of entities, solve physics, process the AI, calculate object occlusions, do the path finding and many other tasks. All of that not only under one second, but most of the time under 16 milliseconds. If we use this as a standard, performance of almost any application is unacceptable.
And websites? This is where we lost battle already. We are wasting so much resources, time and bandwidth by downloading all those interactive video advertisements and animated banners. Maybe we are used to it but just visit a dev.to and imagine each website has same response times. We dreamed about faster websites when internet was a new thing. We still have same dreams now, when fast internet connection is relatively available.

As you can probably imagine, I'm not very happy with the current state of software, but there are always exceptions. I'm using several development tools with acceptable performance and some of them are even developed for free. That means creating fast and responsive software is achievable under any condition. It's all in our hands.

Our team
OUR SCRUM MASTER

August 24, 2018

Continuous Integration (CI)

Let's face the Continuous Integration the development practice that requires developers to integrate code into a shared repository at least once a day per developer. Each check-in is then verified by an automated build, allowing teams to detect problems early. The "continuous" has meaning of regular work, like that you can detect errors quickly, locate them more easily and remove.

Also, it is about verifying if the new code you just wrote broke or not the code that was already working, since the automated tests and other tasks (like syntax verification) are executed when integrating the code. You can't, however expect continuous Integration to get rid of bugs.

Another very important thing when talking about CI is that it needs to be supported by a suite of automated tests (not only unit tests, but also by integration tests, and even better, if possible, by end-to-end tests)

The best part is that continuous Integration is cheap. Not integrating continuously is expensive. If you don’t follow a continuous approach, you’ll have longer periods between integrations. This makes it exponentially more difficult to find and fix problems. Such integration problems can easily knock a project off-schedule, or cause it to fail altogether.

Continuous integration is composed of some essential tasks as Matthew Setter talks about it. you can look them up, but I would like to have a look at the most critical one's.

Make your build self-testing
A self-testing process is the kernel of continuous integration. The build has tests that validate the software. No matter whether you use BDD (Behavior Driven Development), TDD (Test Driven Development), or any of the other xDD’s, testing needs to be front and center in the build process.

Automate the build
Automating the build builds on the fact that it is self-testing. You have the tests in place, now make sure they are run every time. This is a natural complement to software validation.

Make it transparent
The software’s tested before it’s deployed. The deployment happens the same way every time.

Test in a clone of the production environment
This highlights a challenge that has plagued web-based applications for some time. Speaking from personal experience, whether developers develop on Linux, use OSX, or Windows, they usually host on Linux. Even when we develop on the same platform, we may not consider library versions, the existence of extensions, or the extensions’ versions, which can cause problems. So many things can go wrong after the application’s deployed.

Make it easy for anyone to get the latest executable
No matter whether it’s a senior or junior developer, whether it’s a long-term employee or someone brand new to the company, getting a working build of the latest copy of the application or service should be child’s play.

CI is not just a development practice by itself, it has also meaning for being competitive in the market,. It is very good, if you can launch new features that matters for your users faster than your competitors, so you can have advantage and better time to market. Then, CI allows you also to do another very important task from these days, that is called continuous delivery.

Our team
WRITTEN BY VERČA

August 22, 2018

Agile in a kitchen or design?

My two favourite topics just next to the technology are food and design. I love both and I think about both of them most of the time. Therefore, a question appeared in my head: "Does Agile  works just for a software development or is it a way of workflow management also in any other areas we will choose?

Agile in kitchen

Once sitting in a Viennas' restaurant on a meeting brought me to a topic. Can you apply agile framework somewhere between deliciously looking food on a plate and rushing chefs preparing the food? I think it sounds pretty funny, but I do believe you can. In a discussion we had, we grabbed this idea and started going for it.

I have to agree, it was fun to think out of a box in this agile implementation. Still, the conclusion we got to is following. Backlog would be the papers you stick on the wooden beam for the chef to work them off. Of course, the chef relies on given priority of the order. It means person creating chef's backlog knows perfectly the orders and has the ability to organised them accordingly. He might be a Scrum master of your kitchen. As the process is optimized all the food delivery to your customer is once again faster, chefs/waiters/customers less stressed and restaurant is making for sure a bigger profit.

Agile in design

As agile processes can be implemented within any workflow, obviously with a bit of effort. It comes in one's mind to implement it in Design as well as it can be very effective for websites and apps. For instance, you may begin from a user persona that you’ve created, outlining the needs of your target user and using that to branch out and identify the features required. In many organizations, designers span multiple teams (or even products). However, an iterative workflow will see you sitting next to the developer and working in tandem to achieve each iteration as you go.

Understanding the collaborative working style and learning how to estimate will allow you to operate more effectively within a design team. And then, after all, there is Lean UX vs. Agile UX. Many argue that Lean UX is a meaningless term, that doesn’t differ from much older Agile UX and well, it is that way. Exactly, as it is said Lean UX describes methods and their practical application in dynamic environment of a Lean Startup. Agile UX describes update of Agile Software Methodology with UX Design methods. The ultimate goal of Agile UX is to unify developers and designers in the Agile process of product development. Interestingly enough most of the Lean UX teams will actually use Agile UX to coordinate their software development.

So, stay open-minded. Agile is not just for software development? It can be implemented almost anywhere, even in marketing. But let's talk about this topic later.


Our team
WRITTEN BY VERČA

August 12, 2018

The curse of string distance

When you work for some time on a larger project, you may realise it doesn't perform as well as it did. That is usually expected but sometimes, it just doesn't feel justified.

This was also our case and because the performance of the project was important, we had to get our hands dirty with profilers and debuggers. Of course, we found a few bottlenecks.

The cursed method

Pobody's nerfect, but I don't want to write today about them. I want to write about a cursed method, which was with us right from the start. Which was becoming more and more demanding as we were adding more and more classes.

The curse started from an innocent need.. Let's have short field names in Mongo to have organized layout. Let's have self explanatory bean names in application. Spring very helpfully provides the annotation @Field, which can be used to define different database field name than bean field name. And we used it generously. Just a quick search shows about 200 annotation occurrences.

We found out from profiling, the application is spending unhealthy amount of time in method org.springframework.beans.PropertyMatches.calculateStringDistance. Every field, which had different name than Mongo field, calculated the distance against every other bean field.

To make it worse, the application is processing entities from the database and we had to load about 100.000.000 of them. As you can imagine, the number of calculateStringDistance calls was pretty high. At the end of the day, we implemented a cache in PropertyMatches, but now we have to maintain our custom build of spring-beans. Not something we want to do for the whole life span of project.

The more we looked at the issue, the more we believed the call is not necessary. The distance was calculated for exception message of PropertyReferenceException from Spring Data MongoDB project.  I would argue that a preparation of Exception message should be as light as possible. Still, it can be justified if the message is helpful. This exception is caught in QueryMapper.getPath and method returns null, so we even cannot see the content of message, which drags down the performance. However, the field is still mapped correctly probably (this is where we stopped debugging) using the name from annotation @Field. The existence of @Field  itself suggests the bean name will most likely differ from Mongo field name. The PropertyReferenceException shouldn't be needed in those cases.

Is it something that should be investigated in Spring Data MongoDB? Or is there something we should do on our side to prevent this issue happening? There are not a lot of places in our code where to do things differently though. Our bean fields uses @Field annotation and we don't even have a custom bean converter. This is how we get the entity from database.

MongoTemplate template;
public OurEntity ConvertBsonDocument2OurEntity(Document entityObj) {
}

It's hard to believe we are the only ones facing this issue. If you have some observation or solution, feel free to use our comment section. We also opened the ticket DATAMONGO-1991 in Spring Jira and it would help if more people participate in the discussion. It could be hopefully resolved sooner with more feedback.


Our team
OUR SCRUM MASTER

August 2, 2018

Does Scrum save your business?

When I met Agile

I used to have startup developing a mobile application. We were two people working hard and at some point successfully, even though we were a big mess. Both of us doing things from design, to coding, to management, to marketing to planning everything needed. Yes, it is a typical startup mess, that we loved.

agile

But as a co-founder I though of future. Mess between two people is still pretty organised mess, but mess in a growing company with many people? That's deadly.

I had no experience with project management, with being entrepreneur or what the hack is Agile whatsoever. I remember almost one year ago I sat at an interview for an IT company. They loved our app and they supposed I know Agile methodology as it's an obvious thing in development. "We work as agile team, we apply Scrum, Kanban, I think, I don't have to explain what it is. I ended up sitting in front of Google and reading tons of articles about what is agile, who is Product Owner and I came across tons of nonsense as well. At that time I got an offer from a different IT company (oh yeah - this one - you got it right)! and I fell in love with Agile framework - Scrum. Now it's my daily bread and I see - it's not about doing agile, but about being agile.

Scrum saving the whole world?

It is said that above 50% of software projects that were implemented are ending up with over crossing a budget, not delivering products on time or ending up with features nobody ever uses. But why? Why we don't wanna count with the changes? Why we use waterfall? All I ever got from kayaking is once you're in a waterfall, you re pretty much f* and have to fall down until the waterfall hits it's bottom as well as it's in waterfall planning. The changes in half of the process in development are so difficult and so expensive to do, that nobody will rather propose them as it's their own showcase of fail or danger of showing weakness in planning. So, could potentially Scrum save old waterfall driven companies or companies which needs different project management?

It's all about the mindset

No, unless... unless you have the right team with the right mindset. Scrum is not a dogma. It's not a pill you take at night and you are healed in the morning. If you want to apply agile, you have to count with huge loss of people who don't have the mindset, who just can't make it. I was lately part of discussion with Martin Fišer, founder of Brand New- a marketing agency - who turn his company to be completely agile. And yes, it was the case when almost 40 % of people left the work, because they just couldn't make it.Now, question yourself - would you risk that as the CEO of a company with thousands of employees? Imagine an international bank turning agile. Where would you suddenly grab 40% of your team to continue the business?

Still, look around the possibilities to transform your business for better tomorrows. Try piece by piece transform or set your team to agile. It will save you lots of money and lots of energy in future. After all, it's all about the mindset transformation, professionals and the team you built from the scratch. So, before going big or going home, if you have the option - try to built from fundamentals agile team as Spotify did, as Elon Musk does as we do. Your life will be easier and still not sweet too much not to make you work hard.

Our team
WRITTEN BY VERČA

July 22, 2018

Null checking in MongoDB

When we started using MongoDB in our projects, I was very confused by aggregations. But I wrote so many of them at this point, I actually enjoy putting them together. At least I enjoy it more than writing SQL queries. Yet there is one thing I have to process for a while in my mind every time - how to do null checking and field existence checking.

This can be quite confusing especially when you're coming from relational databases.

If we make a connection between table and collection, we see similarities between rows - documents and columns - fields. The biggest difference is, that each document from same collection can contain very different set of fields. So not only the field can contain null, but it may also not be there. Sometimes, we want to differentiate between those. The way of doing checks also depends on the usage and what do you want to achieve.

Query

The syntax of many operators differs when used in query or in aggregation. Null checking in query is rather simple. We can test any field against null. This query will find all accounts not only where field accountId is null, but also where the field doesn't exist.

db.accounts.find({ accountId : null })

To find accounts where accountId does exist and isn't null, we can use operator $ne - not equals. The second query with operator $eq is equivalent of query above.

db.accounts.find({ accountId : {$ne : null} })
db.accounts.find({ accountId : {$eq : null} })

In case we want to check only existence and don't care about null, there is an operator $exists.

db.accounts.find({ accountId : { $exists: true} })
db.accounts.find({ accountId : { $exists: false} })

Aggregation

The stage $match used to filter documents works exactly the same way as query.

{ $match: { accountId : null } }
{ $match: { accountId : {$ne : null}} }
{ $match: { accountId : {$eq : null}} }
{ $match: { accountId : {$exists : true} } }
{ $match: { accountId : {$exists : false} }

So far so good. Here comes one tricky point. In some cases, you don't want to exclude documents from pipeline but rather create a field with value which depends on the existence or null. Using the previous knowledge, we could put together Project stage with null test in condition.

{ $project: { accountIdFlag : {$cond : [{$eq : ["$accountId", null]}, 0, 1]} }}

My goal here is to create field accountIdFlag which will be 0 if accountId is null or non-existent and 1 if it contains value. However if you run this aggregation, the field will always be 1. We have to choose different strategy in this case.

{ $project: { accountIdFlag : {$cond : [{$eq : [{ $ifNull: ["$accountId", null]} , null]}, 0, 1]} }}

If the expression in $ifNull evaluates to null or missing field, replacement expression is returned - null. If not, the value is returned. We test the result against null and this time, we get correct flag - 0 if the field is null or missing and 1 if not.

At the end of the day, null checking isn't hard. But the use case above shows that it can be sometimes tricky so I hope this article will make it clear for those who struggles with null checks in MongoDB.


Our team
OUR SCRUM MASTER

July 7, 2018

Does a developer must have a pet project?

Having a pet project - a project being developed in your free time - can be a lot of fun. Give me a free week and option between spending the whole time on the beach or possibility to work on my stuff, I will probably take #2. Don't get me wrong, who doesn't like vacation? But when as the idle time raises, I'm getting bored and the itch to work on my pet projects grows. Yet by being a software developer, there is an unhealthy assumption that everyone needs to have a pet project. There are even hiring managers, who won't hire you if you don't have some. It's a badge you're committed to your profession. Right?

Well, not really. Just the existence of this unspoken yet important rule can suck all fun from that. When you're pushed to choose and finish project, it could feel like a work. With the difference you aren't paid for it. Can't your previous work experience and interview itself speak for you better? It's not that a surgeon is doing surgeries on hamsters in his free time to prove something to someone. The artist need to have his portfolio prepared because he can't just go to an interview and spend several hours drawing or prove his abilities from a talk. It's harder for a software developers to maintain a portfolio. You most probably aren't allowed to take some code from your company which represents your skills. Even if you could, a method, class or few lines of code out of context speaks nothing about you. So having a pet project makes sense and it makes sense even more, when you're fresh out of school. But let's be honest, your first contact with hiring manager is through your CV, not a GitHub repository. And even the most impressive pet project won't overshadow the interview.

"Hey, you answered all my questions well and you worked for 5 years in a big and well known company. Sadly, your Github account is empty. I hope the time spent with your kids was more important. Not hired."

Working overtime and then working again at home (in front of PC) is just not healthy. It's not good for your eyes, back, blood pressure nor blood sugar level. Don't feel bad that you don't work after work. Yes, having a pet project can be a lot of fun. It allows you to learn new technologies you wouldn't be able to meet in your job. There is a great sense of ownership as you're the owner of project in most cases. By working for example on a computer game, you can learn a totally different skill set, like sound design, music composing or 3D modeling (please throw away the stereotype that software developer cannot do an art). Not necessary something useful for, let's say, FinTech developer, but it's still a lot of fun and that's what matters if you ask me.

As always, everything is about compromises, this topic included. Not having a pet project can be a disadvantage and spending all free time on them is not good either. Instead of having a bunch of unfinished ambitious projects, try to came with one, small and enjoyable. Maybe even useful. Do you like playing games? Try to learn some game engine and participate in Game Jam - events when individual developers or small groups have to design and make a game in very limited time (often few days). My favorite type of game related development is extending game engine with plugins. The scope of them is usually small, it's something which solves my issue and it can be useful to many more people. Even having a small desktop utility is better than nothing. On the other hand, as praiseworthy as the contributing to community developed open source projects is, it's not very presentable. Try to have at least one project solely developed by yourself.

To conclude my thoughts in a tl;dr manner, don't feel bad for not coding in your free time, if your pet project doesn't bring you a joy, throw it away and think about your mental and physical health.


Our team
OUR SCRUM MASTER

June 21, 2018

Visual client for MongoDB database?

Was looking for a visual client for MongoDB database worth it? Definitely!

Fortunately, I do not belong to those developers who are satisfied just with a console. It might be because I have only started to use Linux actively two years ago (during that time Linux even invade my personal PC in the incarnation of Elementary OS). Or maybe it's because I'm a relatively short time in a position of a developer. Either way, the significant number of views of "How to close Vim" article on Stack Overflow is caused by me.

The moment, I reached the point of working with the MongoDB database, I was sure I had to find a visual client. Since I like to use and develop FOSS software, I've been searching in these waters. The waters are deep and wild, but soon I found a great solution, that I have been already using for some time - the Robo 3T, which was that time called Robomongo.

It does not offer much more than Mongo shell and it might be said that it is just a graphical extension of it. Robo 3T shows you all databases and collections in the list, numbers rows and provides automatic query completion. That makes it easy to work with multiple collections at the same time or offers a connection manager while defining the ssh tunnel directly, which is quite useful. But you can't expect more advanced features. If you want to export to CSV, you have to write your own Javascript function and let Robo 3T run it. Some time ago, I came across small problems with too little font, when the client was running on multiple screens at the same time.

Still, with those little things around it' s a clever helper. I am just a bit worried about his future because the project has gone under 3T Software Labs which works on Studio 3T. Unlike it, Robo 3T is an open source and as I went through some experiences - it seems like the best OSS projects have usually just a few of main developers who leads the direction of how the future of the project ends up. Here we reach our pain point. If an open source is an anarchy, it indeed doesn't end up well. Especially, when these developers are working alongside on a competitive commercial solution. The question arises: "Will these developers be really serious about implementing new features?!"

So far I have the impression that 3T Software Labs will want to make Robo 3T a free and poorer alternative to the 3T Studio, which will also serve as an advertisement for their commercial solution.

Of course, I'm realistic and I'm aware that the development of more advanced features costs money. After all, I decided to try Studio 3T and haven't changed since then. It indeed offers a lot, and much more than we need in our projects so far.

But let's have a look at which features made my work with Mongo easier.

  1. The most crucial one is the aggregation interface on which you can build (what the surprise), aggregation! Studio 3T offers a good compromise between graphic representation and the text representation. The interface is not too cluttered, so for that composing aggregation does not become a clicking hell. On one hand, you still have to write the content manually to the content of the individual phases, but on the other hand, it is useful to have each stage in your own tab, because you can quickly change their order or view the output of one phase from the chain. When you're finished, you can simply export your aggregation to the text. If for some reasons, you have to do changes and find out, you are still not done, you will miraculously import it back.
  2. Another frequently used feature in our company is export of collections, aggregations or queries in different formats. Certainly, I would expect the export and import from the client, but not all of them can handle it, and not all of them can manage it as well as Studio 3T. So far, it hasn't happened that I was unable to export the content exactly as I need it. What a real power is that as you don't waste time on finding a solution or another client that can do so well.
  3. I was also very attracted by the new feature, which I, unfortunately, did not have the chance to use, even though, it would be great for us. It's the automatic aggregation conversion to different languages like Javascript, Java, Python and C #. All of these are supported. It means, if your software consists of components written in different languages, this feature is invaluable. Just select the target language on the tab and it's done!!

Do you also use database clients to solve your projects? If not, find some time to try it out. It's worth it and you will not regret. Of course, one client varies from another client, but at least in our case, Studio 3T has made it way easier for us. What's more, finally I'm having fun with the aggregations.


Our team
OUR SCRUM MASTER