Saturday, September 19, 2015

XDG VS SailfishOS

During development of Quickbar, I happened to stumble upon a small issue related to XDG and mime types. This same issue has been raised over at tjc forums lately, so I got inspired and restarted digging.

The problem : lookup of mime types seem to fail relentlessy; opening files with xdg-open as of now (release 1.1.9.28) opens Documents application, no matter which file is actually opened. "So what", you might ask, "where's the problem, apps on SailfishOS rely on QT. We can open anything we want with QDesktopServices::openUrl". Unfortunately, qt internally (and QDesktopServices::openUrl more specifically) relies on xdg-open. So to investigate this, I followed the xdg-mime trace given in the comments of the tjc post.

On SailfishOS xdg tools refer to mimeapps.list for storing mime type associations; xdg scripts come from the package xdg-utils (in case of mer 1.1.0-rc3, keep in mind this version number cause it's the cause of all evil). This is a weird part, but everything is still correct here; this was quite misleading for a long time, as I believed the problem to be mimeapps.list being used as DEFAULT even outside KDE envs; that held until I stumbled on this post in the freedesktop mailing list, which claims that defaults.list has been declared deprecated and superseded by mimeapps.list. In fact, the specification does not happen mention defaults.list at all anymore. The same issue is raised in different environments as well, for example on  this post of the lxde group
 From the same LXDE group post, it seems  the latest and greatest (yet unreleased) version of xdg on the master contains a fix for this. issue.

Saturday, August 29, 2015

The 4 rules of coding - or, of coding-applied strategic thinking.

Coding is not an easy job. From the perspective of the programmer, Coding is a war. I'll tell you why.
You need to do it right; which means; you need to do it clean; you need to do it FAST; and you - potentially, well, pretty much all the time - absolutely NEED to get it right the first time. There is no second chance. You have time to think on how to do it, sure, perhaps weeks, months, YEARS; but you will get that time only ONCE.

One shot. Perfection.

It's not an easy life being a coder. If you like to write; if you like to think; if you like to plan; you are not good enough. You need to like to write AND to think AND to plan. You need a STRATEGY.

Remember, practice makes the master. FOUR simple "riddles" govern the practice of coding, each of them require solving for the specific problem they are being applied on. The four riddles of coding, for properly solving a given problem, are for you find out:
  1. WHAT needs to be done
  2. HOW to achieve that (functionality-wise)
  3. WHY it needs to be done like this (in a structural context)
  4. HOW IT CAN BE USED
Point 1. Is pretty straightforward, and stems off from the requirements.
Point 2. is purely logic; and means collecting the idea of how to do things (i.e. opening files, writing to them vs opening pipes and/or writing to a REST Api etcetcetc). This is the threshold of AVERAGE programmers. They will write average code. This is just a fact. It is not BAD code per se; as it will do what it is meant to do. Only; people working with that code will make the coder's ear ring with F*U's for the rest of his life.
Point 3. Adds perspective in the mix. You might say, "of course I know WHY it needs to be done like this; to fulfill the requirements". That's only half the coin, man! This WHY is waay more subtle and entwingled into the deeper meaning / structure and is meant to get you to think of the solution you are applying (to the problem given by the requirements) in a broader spectrum; or, how more "advanced" programmers like to put it, in terms of "architecture", and in a wholistic view. In short, this point opens up for future usability of your code (funcionality-wise). Things like code structure, Functionality-grouping, application of patterns fall into this area. You can discern Good coders from the rest by the fact that they make use of these tools or not.
Point 4. is the hardest one, and usually only Great coders make it all the way up to this point. This is strategy to it's finest. Point 3 at the n'th potential. Code that is written needs to be USED. Needs to be relied on. If code cannot be used, it is useless, rusty, unreliable. If your code is not "level 4" code, it won't survive a new generation of programmers in your team. "The old code is shit, let's rewrite it", they will say. NIH sindrome.

Thursday, March 12, 2015

Raspberry Pi 2 rocks! mount.cifs not so much..

I got this new toy yesterday and i L-O-V-E it!

One problem tho.
I also have a B+ model, which I used to run owncloud server (plus other sensors servers).
Having collected a decent ammount of data from my sensor network already, I wanted to keep it.
So i upgraded from RPi B+ to RPi2 following instructions on the RPi website.
This went fine & smooth, altho the owncloud server could not anymore connect to the backend NAS (which is used to store all the data).
Symptom of the failure was error -13 upon mounting. Which I figured should translate "in password being wrong" (at least in my case). A few tries with different users (defined on the NAS) didnt give expected results, until I realized to check the version of cifs. Which on the RPi2 is 5.5, and on the nas is 5.1.
Well, it appears the reason for it is that the default security mode of cifs has changed. It used to be ntlm, but it is not anymore - although in the documentation it clearly states it:
 Â·   ntlm Use NTLM password hashing (default)
The above is what comes out of the mount.cifs man page.
Forcing the sec to ntlm in the samba options of the mount solves the issue.

Friday, February 27, 2015

Postmortem : So long, Nokia Store (ex Ovi)!

-Today, Leonard Nimoy died. He undoubtfully represented and played one of the most intriguing characters of Show business history.


Today, after a nice sauna and realxation, I remembered that something else was supposed to die - the Nokia store. I recalled some email sent from Microsoft stating everything would have been shutted down by end of February 2015. Puzzled if it already happend, I grabbed my laptop and quickly checked store.nokia.com. It seemed to still be there. Nokia Store. The place of shattered dreams. Once the biggest online app store by far, with download stats even Apple could have only dreamt of. Nowadays a sad, forgotten place full of outdated software. I recall my software I placed up there. Of course, as everything, I am always late - and as such, I started at the end of Nokia - time i.e. when the N9 was released. The iPhone-killer. The beast that would have reneved Nokia. Well, we all know how well that went.
At that time, with a steady job, and wanting to focus more on Linux, I bought the only barebone-linux phone available at the time, namely the N9. I wanted to start developing for it, I wanted to experiment, learn, investigate possibilities, have an adventure! And where else if not on Nokia (at that time still) Ovi Store? So I grabbed some extra money and funded myself for a shiny new Cyan N9 16 GB (yes back then it was all I could afford). I must admit, thinking about it, even today, I miss the slickness and the particular color and shape. It fitted perfectly in the pocket. But lets not disgress.
On the day the phone was made available, I was at Verkkokauppa, Helsinki, to pick it up, excited about this new adventure. I recall one of the store minions holding a bunch of them in his hand, portraiing an annoied look on his face like for saying : just please get these things out of my hands.
Well, turned out I did the right thing to show up @ Verkkokauppa, cause the Cyam phones got sold out in about a few weeks. The real reason for this was of course shortage of production (which by then had already been stopped by Nokia), but what would I have known back then?
I went back to my work place, and straight away placed the phone on, and started to discover it.
Finding the tools seemed to require a little bit of digging already back then, but from Nokia main page you would be able to gather all the necessary assets. After setting up everyting (not with a few painful mistakes at first), I was ready to go.

My first App was BWizz, an application which basically could be used to edit and import / export bookmarks between the different browsers available for the phone. It was my first Qt App, and as such, a fresh start, especially with qml; but becoming aquainted to it did not take long. Soon enough I was coding plugins for image ressources and UI interfaces like a pro. Qml is that easy, really.
So I got BWizz up to shape for publishing. I knew Nokia had QA process in place; and as I had previous experience with QA processes from my workplace, I knew it would take some time. I did not have many hopes to be able to pass it; but after a few weeks of rejection, eventually I managed to get BWizz published. The pricepoint was #1 (=1 €). "So", i thought, "this is done!" - and went off for a celebration beer. On the first day, BWizz made 20 downloads. I got about 5 emails with questions, and two with more feature requests. In order to try to improve downloads, a free version followed.
Eventually, the total would have added up to 2000 BWizzFree downloads, and about 151 paid BWizz apps. So, 151 €? Hell no! Keep in mind out of revenue made for selling the apps, you need to deduct operator and service provider (aka Nokia) fees. For each BWizz sold, then, 0.7 € would land in my pocket. Peanuts, really.


Monday, January 12, 2015

Find the Fun in "How difficult can debugging mem leaks on mobile be?"; Or : an afternoon spent with Valgrind!

So!

QuickBar on SailfishOS has come a decent way, I would say. We've got :
  * full apps support (both android and native);
  * device-lock screen support;
  * Swipe-launch
  * on-lockscreen presence
  * favourites.

And we have more to come:
  * autostart
  * integration with event screen
  * ...

All in all, a nice (still little) stash of features!
Recently I have been wondering; given SailfishOS's problem with memory management, how does Quickbar's memory footprint affect the overall performance of the system? The application is omnipresent  once launced, and farily easily interacted with. Consequently, a memory leak might threaten stability of the system quite severly if left to run on it's own unnoticed.
Given this thought, I decided to focus on memory consumption next. So, I went off into this investigation by first figuring out the overall size of the memory footprint of QuickBar. There is this utility, pmap, which perfectly fits this purpose. So, devel-su/pkcon install pmap, pmap -x .Upon the first result, I jumped on my chair : Quickbar used a whole whopping 130 MB!! 

Subsequent measures re-confirmed this number. So somewhere memory is either leaking very badly, or ressource usage is not optimized at all. This needed to be investigated more in detail, which led to the hunt for more to this purpouse suitable tools.
The search lead me to Valgrind, open-source tool with excellent run-time analysis toolkit. I fired up a putty term, devel-su / pkcon install valgrind, crossed my fingers, and to my surprise the tool is present in Mer, so it got installed and I was ready to go. First thing to do, was to gain more knowledge on the command line. For my specific purpouses (memory leakage), I came up with the following minimalistic command line:

   --workaround-gcc296-bugs=yes       Ignore a Known compiler issue
   --show-leak-kinds=all                       Display all kinds of leaks
   --log-file=/tmp/output-valgrind.txt   Logfile location
   --leak-check=full                               Check for all possible memory leaks
   --tool=memcheck                              Toolkit for memory check

As a test round, I built a standard Sailfish app in default configuration (meaning debug mode, no optimizations, QML debugging on), deployed it (and the relative debug symbols) to the device. pmap for this empty app gives a default memory footprint of 127MB (which relaxed my nerves a bit). So then next was the run it in Valgrid (BTW, All the type of analysis provided in the trace log are explained down to the detail in the Valgrind manual - brace for long read).

Running the commandline above against my emptyapp test application, gave me a typical recap for the result of the analysis in question (given the due time spent waiting due to the performance impact):
==8455== Memcheck, a memory error detector
==8455== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==8455== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==8455== Command: emptyapp
==8455== Parent PID: 8454
......
==8455== LEAK SUMMARY:
==8455==    definitely lost: 409 bytes in 31 blocks
==8455==    indirectly lost: 46,302 bytes in 500 blocks
==8455==      possibly lost: 173,058 bytes in 1,344 blocks
==8455==    still reachable: 1,384,442 bytes in 16,850 blocks
==8455==         suppressed: 0 bytes in 0 blocks
==8455==
==8455== For counts of detected and suppressed errors, rerun with: -v
==8455== Use --track-origins=yes to see where uninitialised values come from
==8455== ERROR SUMMARY: 3717 errors from 35 contexts (suppressed: 0 from 0)
Notice how Valgrind gives you the option to (tentatively) track origins of memory leaks (not used in this case). Entries in the tracelog file are for example:

==8455== Invalid read of size 4
==8455==    at 0x401A8BC: ??? (in /lib/ld-2.15.so)
==8455==  Address 0x78c52dc is 52 bytes inside a block of size 54 alloc'd
==8455==    at 0x4838CBC: malloc (vg_replace_malloc.c:291)
==8455==    by 0x4008B0F: ??? (in /lib/ld-2.15.so)
==8455==
==8455== Syscall param sendmsg(msg.msg_iov[0]) points to uninitialised byte(s)
==8455==    at 0x554D544: sendmsg (in /lib/libpthread-2.15.so)
==8455==  Address 0x78c9dde is 4,158 bytes inside a block of size 16,424 alloc'd
==8455==    at 0x4838CBC: malloc (vg_replace_malloc.c:291)
==8455==    by 0x7F29E53: ??? (in /usr/lib/libwayland-client.so.0.1.0)
==8455==
==8455== Conditional jump or move depends on uninitialised value(s)
==8455==    at 0x7CA7988: mmap (in /system/lib/libc.so)
==8455==
==8455== Syscall param mmap2(length) contains uninitialised byte(s)
==8455==    at 0x7C9AAD8: __mmap2 (in /system/lib/libc.so)
==8455==
==8455== Syscall param mmap2(offset) contains uninitialised byte(s)
==8455==    at 0x7C9AAD8: __mmap2 (in /system/lib/libc.so)
==8455==
==8455== 20 bytes in 1 blocks are still reachable in loss record 138 of 538
==8455==    at 0x4837D5C: operator new[](unsigned int) (vg_replace_malloc.c:378)
==8455==    by 0x7E2F131: std::_Locale_impl::make_classic_locale() (in /system/lib/libstlport.so)
==8455==
....
==8455== 20 bytes in 1 blocks are still reachable in loss record 139 of 538
==8455==    at 0x4837D5C: operator new[](unsigned int) (vg_replace_malloc.c:378)
==8455==    by 0x7E2F141: std::_Locale_impl::make_classic_locale() (in /system/lib/libstlport.so)
==8455==

.....
==8455== 2,560 bytes in 2 blocks are possibly lost in loss record 468 of 538
==8455==    at 0x4837D5C: operator new[](unsigned int) (vg_replace_malloc.c:378)
==8455==    by 0x7E3B4A5: std::__node_alloc_impl::_S_chunk_alloc(unsigned int, int&) (in /system/lib/libstlport.so)

......
==8455== 46,430 (128 direct, 46,302 indirect) bytes in 1 blocks are definitely lost in loss record 533 of 538
==8455==    at 0x48365A8: calloc (vg_replace_malloc.c:618)
==8455==    by 0x7F01437: xkb_state_new (in /usr/lib/libxkbcommon.so.0.0.0)
==8455==
==8455== 59,204 bytes in 1,092 blocks are possibly lost in loss record 534 of 538
==8455==    at 0x4838CBC: malloc (vg_replace_malloc.c:291)
==8455==    by 0x51E69D7: QArrayData::allocate(unsigned int, unsigned int, unsigned int, QFlags) (in /usr/lib/libQt5Core.so.5.2.2)
==8455== 

We have uninitialized values on conditional jumps and syscalls(not much we can do about these) ;
the "still reachable" data's are - in a very simplified view - mostly pointers not nullified (i.e. pointing to void) or pointing to non-standard locations (cookies before the memory array, some compilers use them to store information like the length of the memory array etc).
The ones you want to keep an eye upon are the "possibly lost" and "definitely lost" ones, which pin-point possible and almost certain memory leaks respectively.

The memory foot print of the same empty app in Release mode (obtained via pmap ) returned a memory footprint of also 127 MBytes.

So, the question arised - if the footprint of a bare-bone Qt app is about 127 MBytes, how much is one without Qt? I headed back to my QtCreator, and built a non-qt c project (containing the default "Hello world" plus a sleep for 10 minutes); then deployed it to the phone, and ran a pmap check on it. The result on the first run (with no modifications to the project) was staggering : 87 MBytes!
Looking at the output of pmap, I discovered the one to blame, namely libQtCore, which was loaded as well. I decided to google around, and stumbled on this post, which seemed to hind to the fact that QtCreator automatically linked QtCore and QtGui even in non-qt projects unless told not to do so (in my case via a QT -= core gui in the .pro file).
So I went back, and added the line above to the project file, recompiled and copied over to the device, and re-run. The result was a significantly reduced memory foot print, which was between 2 and 3 megabytes.