fbpx
Tag

Software

October 16, 2018

Is software getting worse?

My question targets not only desktop software but also mobile apps, web applications and websites. Seems like software is getting worse and it won't be better. And I just have to start my rumble with the worst - mobile apps.

A very good example is Tinder. This thing needs almost 25 seconds to load and takes 50MB of space. The usability of profile page is very bad (oh, you don't need to sort your photos), application sometimes don't want to load profile pictures and transitions between pages are lagy even when the application is fully loaded in the memory. And we are talking about something, what is basically a chat (not even a good one).
You would imagine that with such high earnings, company would be able to easily craft better user experience and improve the performance. But why, when new phones with better hardware are released constantly? We are getting better hardware, but sadly the user experience is still same because the software is getting slower. After few Android updates, I can install only few applications because the internal storage can barely fit updated OS. Even though my phone is far superior than fist version of Samsung Galaxy S, the loading times are somehow slower.

Desktop software isn't better. When I tried to install Visual Studio last time, it took forever to download and install several gigabytes of data. The size of whole package can be for sure explained, but it still surprises me how can a software package take so much space. We aren't talking here about computer game, where executable is several times smaller than resources like textures, sounds, music and 3D models. It also contrasts against exceptions, which often comes from open source territory. These exceptions can offer minimal footprint and rich features packed in single executable file. Visual Studio also install several gigabytes of itself on system drive, even though you selected different hard drive. Not nice, when your system drive is SSD with limited space.
Another concerning trend is rising usage of web technologies in desktop applications. When I saw an Atom text editor for the first time, I was excited to try lightweight alternative to bloated (don't mean it in a bad way) IDEs for coding. The editor is indeed faster comparing to full fledged IDE, but it's hardly lightweight and smooth user experience. When you compare it against C++ based Sublime Text, the difference is noticeable. I'm scared of the day when web technologies will take over the desktop environment. We are slowly getting there. Sooner or later, we will start seeing advertisements in the software we daily use. Heck, Windows 10 is already doing it.

Someone can see a poor performance as a non-issue. You could even say that most desktop apps are pretty fast. But let's take a look at video games. They have to do a lot of things in one second - handle tens or thousands of entities, solve physics, process the AI, calculate object occlusions, do the path finding and many other tasks. All of that not only under one second, but most of the time under 16 milliseconds. If we use this as a standard, performance of almost any application is unacceptable.
And websites? This is where we lost battle already. We are wasting so much resources, time and bandwidth by downloading all those interactive video advertisements and animated banners. Maybe we are used to it but just visit a dev.to and imagine each website has same response times. We dreamed about faster websites when internet was a new thing. We still have same dreams now, when fast internet connection is relatively available.

As you can probably imagine, I'm not very happy with the current state of software, but there are always exceptions. I'm using several development tools with acceptable performance and some of them are even developed for free. That means creating fast and responsive software is achievable under any condition. It's all in our hands.

Our team
OUR SCRUM MASTER

October 1, 2018

Mongo security

Sooner or later it is unevitable to secure your database. At least in my opinion... It was the first time I was securing a MongoDB instance and when I was looking for some information I came across this blog post. At first, if you want to set up username&password authentication for your MongoDB instance, you will find the article really helpful. Secondly, you can take much more from the post...specifically almost 600TB of data from all around the world (if you wish of course).

MongoDB security

So...I promised a remarkably big bundle of data. Good news is it is as easy as copypasting a command to your console to get it...if you have 600TBs of storage space. If you want to try this out, just read this analysis mentioned in the blog referenced above. The bad news is that probably almost none of those hundreds of whatever bytes out there are publicly accessible intentionally. This data comes from more than 30 000 completely unsecured MongoDB instances. It is natural to ask for a reason - so why is it that this big number of MongoDB instances serves data to just anyone who asks for it?

At first, it is important to note that 30 000 IS a big number. The main reason behind this global security neglection is simple - for a fairly long time, default MongoDB configuration was left completely unsecured.

Security is usually the first argument against using Mongo and it has been heavily criticised in that field (especially after a lovely global ransomware attack. But MongoDB developers are definitely not the only ones in charge for the situation - in fact, those 30 000 instances is how 'nah, it'll be OK' looks like - thousands of people just didn't want to spend a while configuring even a simple authentication mechanism. Simply the idea that your data is safe because it is not any kind of top secret information with a little bit of natural human laziness grant results (almost 600TBs of results).

After I realised how simple it is to provide the basic authentication settings I was just wondering why so many people haven't done these simple steps. In my opinion the biggest problem with data security is the perception of data - sadly not everyone sees it as something valuable nowadays...but as I would not leave alone my wallet or phone I would not do the same with data. The second important factor is our human nature - everyone has this 'nah, it'll be OK' in them. For me the most important message is that setting up at least username&password authentication for your MongoDB is a small step for a developer, but a huge leap for the security of your data.

Our team
WRITTEN BY VÁCLAV

September 3, 2018

Coalescence of Business & IT and the delayed train

I was wondering since I am in a business and since I am in an IT, why these two sectors were seen separately for such a long time. It seems last few years people started realising they can work together. Nowadays, they are many companies based simply on the digital products, look at Airbnb, Uber, Spotify.. without the technology and some algorithms plus great programmers they are nothing.

But! Let's stop for a moment there. These companies grabs the best of IT, technology world and base their Business on a digital world. How is it with companies providing services though? For instance banks, insurance companies, city transport state companies, railway companies and many others. Are they using technology? Are they using it effectively and do they even know the possibilities? Sometimes, as we all know, it is way more about "do they want to know the possibilities of optimisation of the business or it's just all working fiiiiine for them without trying to find new ways?". I have seen many times as companies, startups or individuals are jumping for more of the technology, when they are getting in a great lost, when they are desperately trying to save the business. But shouldn't it be the other way? Technology helping to grow the business as soon as possible? Providing to people the best service and to the company a great revenue asap?

I remember some months ago I was at a Hackathon for Ostrava transport and I realised how the company works. How the people works. You see, Ostrava has one of the most optimised city transport (which I would not guess until I've seen it) and one of the most modern one (you could buy a ticket in a tram/bus by card way before you could do in Prague, which is kind of ironic as it is the main city full of tourists.^^) Why in Ostrava? Because they are people who wants to change things, because there are people trying to move the mountain doing hackathons, educating themselves about datas and new technologies. Indeed, crazy ideas (in the best meaning of the word) were presented in the hackathon as a face-recognition machine learning to say, if people are satisfied with the services or data visualisation of people using the transport. That could lead to optimised timetable of tram lines where is a very low traffic and vice versa to support transport lines with a big amount of people trying to get from a point A to a point B. And it is so simple, all the people attending the hackathon tried to help with the technology and find the best solution by their own approach. Thumbs up for that.

The same should work for the banks, the insurance companies and all great corporates. They all seem to have internal IT team. We had lately discussion with a friend of mine, if these IT teams have the same range of visions and passions for the technology and with that bringing the quality and freshness. It is indeed so hard to accept the modern technology, the growth or using open sources instead of heavy big old softwares - saving thousands of Euros just with this little change? Or even just having a different project management than the waterfall. Fortunately, there are companies, which are trying to find the way to provide better, faster and cheaper service with a smart usage of the IT, softwares, BI and other technologies.

Still, they are some others, that needs to optimise way more and maybe have different approach to things. Because maybe, if they are willing to use more of the modern technology, approaches.. I would not sit in an hour delayed train on 2,5 hours journey again and again. So, maybe let's do a hackathon to bring brainstorm ideas on how to provide a better service to the travelers.

Our team
WRITTEN BY VERČA

August 24, 2018

Continuous Integration (CI)

Let's face the Continuous Integration the development practice that requires developers to integrate code into a shared repository at least once a day per developer. Each check-in is then verified by an automated build, allowing teams to detect problems early. The "continuous" has meaning of regular work, like that you can detect errors quickly, locate them more easily and remove.

Also, it is about verifying if the new code you just wrote broke or not the code that was already working, since the automated tests and other tasks (like syntax verification) are executed when integrating the code. You can't, however expect continuous Integration to get rid of bugs.

Another very important thing when talking about CI is that it needs to be supported by a suite of automated tests (not only unit tests, but also by integration tests, and even better, if possible, by end-to-end tests)

The best part is that continuous Integration is cheap. Not integrating continuously is expensive. If you don’t follow a continuous approach, you’ll have longer periods between integrations. This makes it exponentially more difficult to find and fix problems. Such integration problems can easily knock a project off-schedule, or cause it to fail altogether.

Continuous integration is composed of some essential tasks as Matthew Setter talks about it. you can look them up, but I would like to have a look at the most critical one's.

Make your build self-testing
A self-testing process is the kernel of continuous integration. The build has tests that validate the software. No matter whether you use BDD (Behavior Driven Development), TDD (Test Driven Development), or any of the other xDD’s, testing needs to be front and center in the build process.

Automate the build
Automating the build builds on the fact that it is self-testing. You have the tests in place, now make sure they are run every time. This is a natural complement to software validation.

Make it transparent
The software’s tested before it’s deployed. The deployment happens the same way every time.

Test in a clone of the production environment
This highlights a challenge that has plagued web-based applications for some time. Speaking from personal experience, whether developers develop on Linux, use OSX, or Windows, they usually host on Linux. Even when we develop on the same platform, we may not consider library versions, the existence of extensions, or the extensions’ versions, which can cause problems. So many things can go wrong after the application’s deployed.

Make it easy for anyone to get the latest executable
No matter whether it’s a senior or junior developer, whether it’s a long-term employee or someone brand new to the company, getting a working build of the latest copy of the application or service should be child’s play.

CI is not just a development practice by itself, it has also meaning for being competitive in the market,. It is very good, if you can launch new features that matters for your users faster than your competitors, so you can have advantage and better time to market. Then, CI allows you also to do another very important task from these days, that is called continuous delivery.

Our team
WRITTEN BY VERČA

August 22, 2018

Agile in a kitchen or design?

My two favourite topics just next to the technology are food and design. I love both and I think about both of them most of the time. Therefore, a question appeared in my head: "Does Agile  works just for a software development or is it a way of workflow management also in any other areas we will choose?

Agile in kitchen

Once sitting in a Viennas' restaurant on a meeting brought me to a topic. Can you apply agile framework somewhere between deliciously looking food on a plate and rushing chefs preparing the food? I think it sounds pretty funny, but I do believe you can. In a discussion we had, we grabbed this idea and started going for it.

I have to agree, it was fun to think out of a box in this agile implementation. Still, the conclusion we got to is following. Backlog would be the papers you stick on the wooden beam for the chef to work them off. Of course, the chef relies on given priority of the order. It means person creating chef's backlog knows perfectly the orders and has the ability to organised them accordingly. He might be a Scrum master of your kitchen. As the process is optimized all the food delivery to your customer is once again faster, chefs/waiters/customers less stressed and restaurant is making for sure a bigger profit.

Agile in design

As agile processes can be implemented within any workflow, obviously with a bit of effort. It comes in one's mind to implement it in Design as well as it can be very effective for websites and apps. For instance, you may begin from a user persona that you’ve created, outlining the needs of your target user and using that to branch out and identify the features required. In many organizations, designers span multiple teams (or even products). However, an iterative workflow will see you sitting next to the developer and working in tandem to achieve each iteration as you go.

Understanding the collaborative working style and learning how to estimate will allow you to operate more effectively within a design team. And then, after all, there is Lean UX vs. Agile UX. Many argue that Lean UX is a meaningless term, that doesn’t differ from much older Agile UX and well, it is that way. Exactly, as it is said Lean UX describes methods and their practical application in dynamic environment of a Lean Startup. Agile UX describes update of Agile Software Methodology with UX Design methods. The ultimate goal of Agile UX is to unify developers and designers in the Agile process of product development. Interestingly enough most of the Lean UX teams will actually use Agile UX to coordinate their software development.

So, stay open-minded. Agile is not just for software development? It can be implemented almost anywhere, even in marketing. But let's talk about this topic later.


Our team
WRITTEN BY VERČA

August 12, 2018

The curse of string distance

When you work for some time on a larger project, you may realise it doesn't perform as well as it did. That is usually expected but sometimes, it just doesn't feel justified.

This was also our case and because the performance of the project was important, we had to get our hands dirty with profilers and debuggers. Of course, we found a few bottlenecks.

The cursed method

Pobody's nerfect, but I don't want to write today about them. I want to write about a cursed method, which was with us right from the start. Which was becoming more and more demanding as we were adding more and more classes.

The curse started from an innocent need.. Let's have short field names in Mongo to have organized layout. Let's have self explanatory bean names in application. Spring very helpfully provides the annotation @Field, which can be used to define different database field name than bean field name. And we used it generously. Just a quick search shows about 200 annotation occurrences.

We found out from profiling, the application is spending unhealthy amount of time in method org.springframework.beans.PropertyMatches.calculateStringDistance. Every field, which had different name than Mongo field, calculated the distance against every other bean field.

To make it worse, the application is processing entities from the database and we had to load about 100.000.000 of them. As you can imagine, the number of calculateStringDistance calls was pretty high. At the end of the day, we implemented a cache in PropertyMatches, but now we have to maintain our custom build of spring-beans. Not something we want to do for the whole life span of project.

The more we looked at the issue, the more we believed the call is not necessary. The distance was calculated for exception message of PropertyReferenceException from Spring Data MongoDB project.  I would argue that a preparation of Exception message should be as light as possible. Still, it can be justified if the message is helpful. This exception is caught in QueryMapper.getPath and method returns null, so we even cannot see the content of message, which drags down the performance. However, the field is still mapped correctly probably (this is where we stopped debugging) using the name from annotation @Field. The existence of @Field  itself suggests the bean name will most likely differ from Mongo field name. The PropertyReferenceException shouldn't be needed in those cases.

Is it something that should be investigated in Spring Data MongoDB? Or is there something we should do on our side to prevent this issue happening? There are not a lot of places in our code where to do things differently though. Our bean fields uses @Field annotation and we don't even have a custom bean converter. This is how we get the entity from database.

MongoTemplate template;
public OurEntity ConvertBsonDocument2OurEntity(Document entityObj) {
}

It's hard to believe we are the only ones facing this issue. If you have some observation or solution, feel free to use our comment section. We also opened the ticket DATAMONGO-1991 in Spring Jira and it would help if more people participate in the discussion. It could be hopefully resolved sooner with more feedback.


Our team
OUR SCRUM MASTER

August 2, 2018

Does Scrum save your business?

When I met Agile

I used to have startup developing a mobile application. We were two people working hard and at some point successfully, even though we were a big mess. Both of us doing things from design, to coding, to management, to marketing to planning everything needed. Yes, it is a typical startup mess, that we loved.

agile

But as a co-founder I though of future. Mess between two people is still pretty organised mess, but mess in a growing company with many people? That's deadly.

I had no experience with project management, with being entrepreneur or what the hack is Agile whatsoever. I remember almost one year ago I sat at an interview for an IT company. They loved our app and they supposed I know Agile methodology as it's an obvious thing in development. "We work as agile team, we apply Scrum, Kanban, I think, I don't have to explain what it is. I ended up sitting in front of Google and reading tons of articles about what is agile, who is Product Owner and I came across tons of nonsense as well. At that time I got an offer from a different IT company (oh yeah - this one - you got it right)! and I fell in love with Agile framework - Scrum. Now it's my daily bread and I see - it's not about doing agile, but about being agile.

Scrum saving the whole world?

It is said that above 50% of software projects that were implemented are ending up with over crossing a budget, not delivering products on time or ending up with features nobody ever uses. But why? Why we don't wanna count with the changes? Why we use waterfall? All I ever got from kayaking is once you're in a waterfall, you re pretty much f* and have to fall down until the waterfall hits it's bottom as well as it's in waterfall planning. The changes in half of the process in development are so difficult and so expensive to do, that nobody will rather propose them as it's their own showcase of fail or danger of showing weakness in planning. So, could potentially Scrum save old waterfall driven companies or companies which needs different project management?

It's all about the mindset

No, unless... unless you have the right team with the right mindset. Scrum is not a dogma. It's not a pill you take at night and you are healed in the morning. If you want to apply agile, you have to count with huge loss of people who don't have the mindset, who just can't make it. I was lately part of discussion with Martin Fišer, founder of Brand New- a marketing agency - who turn his company to be completely agile. And yes, it was the case when almost 40 % of people left the work, because they just couldn't make it.Now, question yourself - would you risk that as the CEO of a company with thousands of employees? Imagine an international bank turning agile. Where would you suddenly grab 40% of your team to continue the business?

Still, look around the possibilities to transform your business for better tomorrows. Try piece by piece transform or set your team to agile. It will save you lots of money and lots of energy in future. After all, it's all about the mindset transformation, professionals and the team you built from the scratch. So, before going big or going home, if you have the option - try to built from fundamentals agile team as Spotify did, as Elon Musk does as we do. Your life will be easier and still not sweet too much not to make you work hard.

Our team
WRITTEN BY VERČA

July 26, 2018

PDI SAS Reader

 

If you work in FinTech, sooner or later (but probably sooner) SAS data set will get into your project. In a field dominated by one system, there is hardly a place for rejection. Unfortunately, for small companies it is very hard to integrate SAS in their stack. In our projects, we use ETL to gather data from different sources. To be more precise, we use open source tool Pentaho Data Integration. Even though it has some flaws, it worked very well for us so far - with one little, yet important exception. The input step isn't able to read compressed SAS datasets. After discussing different options, we opted to roll out our own solution. Starting from a scratch probably wouldn't be worth it and we, fortunately, didn't have to do that. We found Parso library - lightweight open source SAS7BDAT reader.

The API was very easy to learn. On the other hand it took significantly more time to code the plugin itself as we didn't have previous experience with making PDI plugins. Still, it took us around 2 days of pure development time to have something usable and about week of testing, fixing bugs and implementing nice to have features. We just love to use open source software in our projects, therefore it felt right to open source PDI SAS reader too. So if you have any problem or feature idea, please visit our issue tracker on Github.

But enough with history, let's get into those meaty features we implemented. The most significant quality of life improvement is the way how columns are defined. If you don't generate the SAS data set by yourself (but maybe even if you do), it might happen that the order of columns changes. Not surprisingly the reader step is using columns' names to identify fields in stream. This is done only with first row and indices are cached, so the lookup shouldn't have impact on performance. New columns can be added this way anywhere to data set and it won't break your transformation, which is just awesome. The latest feature (in the time of writing this article still in snapshot) is also related to columns. At the start of the project, all columns were mandatory. If any was missing, the error was thrown. This was changed later, thus optional columns were implemented. If the column is flagged, his presence inside the file isn't checked and if missing, field in stream will have null value.

Many other checks were implemented too - missing step connections, accessibility of SAS file, file emptiness check or column presence check. Warnings, errors and additional information about file (number of columns and rows) are shown to the user when using Verify this transformation button. The usability of the step itself was also important topic for us. The output field names can be renamed and values can be converted to desired output format. We thought it would be a good idea to force output format to BigNumber, when using feature Get Fields (to save the mouse from some clicks), so we added the option which will suggest this format instead of Number when checked. It turned out it wasn't needed at all for our use case but you might find it useful for some projects. In fact, we would like to know your opinion about the plugin in general. Give us your input and who knows, we may implement your desired feature!


Our team
OUR SCRUM MASTER

July 22, 2018

Null checking in MongoDB

When we started using MongoDB in our projects, I was very confused by aggregations. But I wrote so many of them at this point, I actually enjoy putting them together. At least I enjoy it more than writing SQL queries. Yet there is one thing I have to process for a while in my mind every time - how to do null checking and field existence checking.

This can be quite confusing especially when you're coming from relational databases.

If we make a connection between table and collection, we see similarities between rows - documents and columns - fields. The biggest difference is, that each document from same collection can contain very different set of fields. So not only the field can contain null, but it may also not be there. Sometimes, we want to differentiate between those. The way of doing checks also depends on the usage and what do you want to achieve.

Query

The syntax of many operators differs when used in query or in aggregation. Null checking in query is rather simple. We can test any field against null. This query will find all accounts not only where field accountId is null, but also where the field doesn't exist.

db.accounts.find({ accountId : null })

To find accounts where accountId does exist and isn't null, we can use operator $ne - not equals. The second query with operator $eq is equivalent of query above.

db.accounts.find({ accountId : {$ne : null} })
db.accounts.find({ accountId : {$eq : null} })

In case we want to check only existence and don't care about null, there is an operator $exists.

db.accounts.find({ accountId : { $exists: true} })
db.accounts.find({ accountId : { $exists: false} })

Aggregation

The stage $match used to filter documents works exactly the same way as query.

{ $match: { accountId : null } }
{ $match: { accountId : {$ne : null}} }
{ $match: { accountId : {$eq : null}} }
{ $match: { accountId : {$exists : true} } }
{ $match: { accountId : {$exists : false} }

So far so good. Here comes one tricky point. In some cases, you don't want to exclude documents from pipeline but rather create a field with value which depends on the existence or null. Using the previous knowledge, we could put together Project stage with null test in condition.

{ $project: { accountIdFlag : {$cond : [{$eq : ["$accountId", null]}, 0, 1]} }}

My goal here is to create field accountIdFlag which will be 0 if accountId is null or non-existent and 1 if it contains value. However if you run this aggregation, the field will always be 1. We have to choose different strategy in this case.

{ $project: { accountIdFlag : {$cond : [{$eq : [{ $ifNull: ["$accountId", null]} , null]}, 0, 1]} }}

If the expression in $ifNull evaluates to null or missing field, replacement expression is returned - null. If not, the value is returned. We test the result against null and this time, we get correct flag - 0 if the field is null or missing and 1 if not.

At the end of the day, null checking isn't hard. But the use case above shows that it can be sometimes tricky so I hope this article will make it clear for those who struggles with null checks in MongoDB.


Our team
OUR SCRUM MASTER

July 7, 2018

Does a developer must have a pet project?

Having a pet project - a project being developed in your free time - can be a lot of fun. Give me a free week and option between spending the whole time on the beach or possibility to work on my stuff, I will probably take #2. Don't get me wrong, who doesn't like vacation? But when as the idle time raises, I'm getting bored and the itch to work on my pet projects grows. Yet by being a software developer, there is an unhealthy assumption that everyone needs to have a pet project. There are even hiring managers, who won't hire you if you don't have some. It's a badge you're committed to your profession. Right?

Well, not really. Just the existence of this unspoken yet important rule can suck all fun from that. When you're pushed to choose and finish project, it could feel like a work. With the difference you aren't paid for it. Can't your previous work experience and interview itself speak for you better? It's not that a surgeon is doing surgeries on hamsters in his free time to prove something to someone. The artist need to have his portfolio prepared because he can't just go to an interview and spend several hours drawing or prove his abilities from a talk. It's harder for a software developers to maintain a portfolio. You most probably aren't allowed to take some code from your company which represents your skills. Even if you could, a method, class or few lines of code out of context speaks nothing about you. So having a pet project makes sense and it makes sense even more, when you're fresh out of school. But let's be honest, your first contact with hiring manager is through your CV, not a GitHub repository. And even the most impressive pet project won't overshadow the interview.

"Hey, you answered all my questions well and you worked for 5 years in a big and well known company. Sadly, your Github account is empty. I hope the time spent with your kids was more important. Not hired."

Working overtime and then working again at home (in front of PC) is just not healthy. It's not good for your eyes, back, blood pressure nor blood sugar level. Don't feel bad that you don't work after work. Yes, having a pet project can be a lot of fun. It allows you to learn new technologies you wouldn't be able to meet in your job. There is a great sense of ownership as you're the owner of project in most cases. By working for example on a computer game, you can learn a totally different skill set, like sound design, music composing or 3D modeling (please throw away the stereotype that software developer cannot do an art). Not necessary something useful for, let's say, FinTech developer, but it's still a lot of fun and that's what matters if you ask me.

As always, everything is about compromises, this topic included. Not having a pet project can be a disadvantage and spending all free time on them is not good either. Instead of having a bunch of unfinished ambitious projects, try to came with one, small and enjoyable. Maybe even useful. Do you like playing games? Try to learn some game engine and participate in Game Jam - events when individual developers or small groups have to design and make a game in very limited time (often few days). My favorite type of game related development is extending game engine with plugins. The scope of them is usually small, it's something which solves my issue and it can be useful to many more people. Even having a small desktop utility is better than nothing. On the other hand, as praiseworthy as the contributing to community developed open source projects is, it's not very presentable. Try to have at least one project solely developed by yourself.

To conclude my thoughts in a tl;dr manner, don't feel bad for not coding in your free time, if your pet project doesn't bring you a joy, throw it away and think about your mental and physical health.


Our team
OUR SCRUM MASTER