This bug took me on an interesting (well…) journey into what cross platform really means.It is about a complex thing like filenames containing special characters, my favorite example is the directory “Übergröße”. Directories and files like this could not successfully be synced with the latest client.

Windows does not utf8 by default. As far as I understood it, it does a local encoding by default. If you use normal C functions to access the filesystem, something like readdir or so, you get the names in that encoding. But, because that caused problems in some point, there is another set of functions which often start with _w (for example _wreaddir) and that has an interface to deal with wide characters. This is the way to internationalization.

So what needs to be done is to use the wide character implementations everywhere where you deal with filenames (in my case). That, in turn, requires to use wchar_t instead of normal char. But that again requires a lot of #ifdef _WIN32, if its done the naive way at every place it needs to be done. I learned that typedef helps here and a meta type is defined called _TCHAR. And also for all the needed functions, meta functions are defined which are replaced by tricky defines depending on the platform. For example _treaddir becomes readdir on linux and _wreaddir on win32, while _TCHAR becomes wchar_t on Win and char on linux. The code is written using the meta types to not poison it too much with platform ifdefs.

But that is only the first step. Wide character is not utf8! And since utf8 should be used within the software all over, a conversion from wide character to utf8 and back is needed whenever the file system is accessed. Thankfully from csyncs nature, these places are quite nicely concentrated.

All that combines into a larger patch. And now the “Übergröße” folder can be synced correctly from Windows to Linux and back. Great.

What can we learn from that? Well, easy, and once again: If you have no Qt, you’re alone. Better don’t let that happen. If you have Qt, be happy and aware of what it does for you

In the next three months I am going to develop a feed aggregator for ownCloud as a Google Summer of Code 2012 (GSoC) project under KDE. The mentors of this project are Frank Osterfeld and Jakob Sack. As you might remember from my previous posts (1 and 2), this belongs to a wider project that aims at synchronizing KDE Akregator on the cloud. If you are interested in more details, you can read my full proposal here.
This is how the web interface of the aggregator is going to look like (in two different views):

There are big changes happening in these days for what concerns the way ownCloud handles apps. Consequences of this are:

• the apps folder can now live outside the main owncloud folder;
• core and secondary apps are going to be moved to a single repository.

If you have already installed ownCloud and you want to follow the development of my app, then you need to clone the following repository:

owncloud/apps.git branch:newsapp (https://gitorious.org/owncloud/apps/commits/newsapp).

A stub of the feed reader app is already in the subfolder news (‘News’ is also going to be the user-friendly name of the app, as well as the owncloud app id – feel free to comment if you like it or not).
Unfortunately it is not possible to specify multiple app folders yet. If you want to both use ownCloud core apps (from the owncloud/owncloud.git repo) and check the development of my app at the same time, I recommend the following setup:

1. install ownCloud in /your_web_server_root/owncloud
2. clone the above-mentioned repository in the folder /your_web_server_root/apps (remember: my app is in the working branch newsapp
3. create a symbolic link pointing to the folder /your_web_server_root/apps/news from the /your_web_server_root/owncloud/apps folder.

Things will become cleaner once the transition to the new app system will be complete. Stay tuned!

Only a couple of weeks after the first release we could come up with a quite long changelog, larger than for a maintainance release.

First and foremost, oCC now supports SSL connections and does not store passwords any more, well, in clear text in a config file. Still there is no strong encryption for the password in (as that’s a non trivial if not impossible thing) but its not clear text any more, so accidents should not longer happen. For those who really care and put security over convenience there now is the option to not store the password at all but provide it at startup. The whole password storage will get more attention soon, I did some work on that already, also using kwallet, more on that in another post.

The whole start process of oCC, which is more complex as it might seem as libcsync expects a nice environment with a verified connection to the ownCloud, was cleaned and matured. I had to learn about QNAM but now we seem to be friends.

Efforts also went into the following areas:

• Error handling: People got confused about error messages from oCC which were not accurate here and than. I added more fine granular error reporting to libcsync so that oCC now can give even more helpful error messages.
• MacOSX: On the sprint weekend in Stuttgart, we made good progress on that platform, will also soon get a release. Domme got the most out of cmake to get dmg image building going, thanks, you rock!
• Qt 4.6: As usual one has to learn that current distributions ship not so current Qt versions. Thanks to Thomas, he put quite some effort in porting back to Qt 4.6 – so that Debians friends can also use oCC now.
• Packages: Check out the community repo on OBS: We now can provide packages for way more platforms including Ubuntu and Debian. Still your help would be appreciated in maintaining.

Speaking about the recent sprint in Stuttgart: It was great to be there, ownCloud has an open, friendly and welcoming community in which you really feel the spirit of working on something new and interesting with cool potential. I really enjoy being here,. Thanks guys for all your work and feedback

In this example owncloud refers to the URL to the root of your ownCloud installation e.g. http://example.com/owncloud, user is your login and password is your password.

The URL to use is the same you use if you want to download the calendar or address book manually. In the upper, right corner of the Calendar and the Contacts app, there’s a button named respectively Calendars and Address books. When you click on that, you get a dialog box with a list of your calendar(s) and address book(s). The screen shot below is from the Contacts app. To download manually push the  button. Instead of doing that, right-click on it, and select “Copy link location” (or whatever your browser calls it) from the context menu.

Your clipboard will now hold a URL looking something owncloud/apps/contacts/export.php?bookid=1 – for the calendar it would look like owncloud/apps/calendar/export.php?calid=1.

Make a simple script using those URLs:

#!/bin/bash
DATE=`date +"%Y-%W"`

# Download and gzip contacts.
wget --auth-no-challenge --no-clobber --http-user=user --http-password=password \
  -O $HOME/pimdata_backup/contacts-$DATE.vcf owncloud/apps/contacts/export.php?bookid=1
gzip -f $HOME/pimdata_backup/contacts-$DATE.vcf

# Download and gzip calendar.
wget --auth-no-challenge --no-clobber --http-user=user --http-password=password \
  -O $HOME/pimdata_backup/calendar-$DATE.ics owncloud/apps/calendar/export.php?calid=1
gzip -f $HOME/pimdata_backup/calendar-$DATE.ics

The --auth-no-challenge is to avoid having the server send an authentication challenge, which would mess up the download. The --no-clobber is to avoid downloading a file that already exists. When gzipping the downloaded file it gets a .gz extension appended to the filename, so it should normally not be an issue though. The backup files will now have names containing the year and week number like contacts-2012-17.vcf.gz and calendar-2012-17.ics.gz. If you prefer to backup in another schedule than weekly, you will of course have to change the date command accordingly.

Save the script somewhere preferably in your PATH – I use ~/bin for my scripts – and make it executable: chmod u+x ~/bin/owncloud_backup.sh – or whatever you choose to call it. Test if the script works and set it up to run on a regular basis as a cron job or any other job scheduler; I use KAlarm because it’s so damn easy to use

You could use curl instead of wget, it’s just a matter of preference.

Update: As Klaus Muth mentions in a comment, it cannot be stressed enough that this method should not be used in a multi user environment as it places passwords in the process list at runtime.

• General stabilization of the ownCloud 4 code base.
• Work on a solution that solves the big file upload problem once and for all.
• Huge database optimizations for better performance.
• Improved KDE integration
• Implementation of an LDAP groups backend and big improvements in the LDAP user integration.
• Implementation of a lucene search engine for files. We now have proper searching. :-)
• UX optimisation all over the place
• Lot´s of small design improvements
• Improvement on ownCloud.org
• Work on Debian Packaging
• OpenID support
• Drag and Drop uploading
• Implementation of a "Find my Android phone" location based feature.
• Working on versioning and encryption support
• Lots of bugfixes at the Desktop Syncing client
• Porting of the Desktop Syncing Client to Mac OS X. Release soon.
• UX improvements at the Desktop Syncing Client.
• Work started on an ownCloud mail client. 
• Lots of bugs fixed

Build it!

Linux and Windows development should go in parallel as easy as possible.

Building CSync

To build mirall, csync must be built first. CSync is hosted in its upstream git repo and there is also my development branch which holds the latest changes.

Clone the csync branch into a directory. In parallel to the cloned csync dir, create a new directory buildcsync as a cmake build dir. Change into buildcsync and call cmake like this:

cmake -DCMAKE_BUILD_TYPE="Debug" ../csync

and watch its output. You probably have to fulfill some dependencies, make sure to install all the needed devel packages. You will need log4c, iniparser, sqlite3 and for the modules libssh, libsmbclient and neon for the ownCloud module. Once cmake succeeds, call make to build it. So far relaxed for Linux.

To build csync for Windows, there are a couple of possibilities. The one I chose was to cross compile with mingw under openSUSE. That way I can build for all on one devel machine under my prefered system.

For that, I installed the cross compile and mingw32 packages from the openSUSE Build Service, which really demonstrates power here. I used the mingw repository. Kudos at this point to Dominik Schmidt, a Tomahawk developer, who helped me a lot to set all up and to all people who work in OBS to maintain the mingw repo.

Basically the cross compiler and libs (eg. packages mingw32-cross-gcc, mingw32-gcc-c++ and mingw32-cross-cpp) and the dependencies for the software to build have to be installed from the mingw repo. An action item is left to dig which in detail.

After installation you should have some mingw32-tools such as mingw32-cmake which should be used to build for win.

Now create a directory win and within that again buildcsync. In there, start cmake with

mingw32-cmake -DCMAKE_BUILD_TYPE="Debug" -DWITH_LOG4C=OFF ../../csync

That should do it. I did not find log4c for Win32, so I disabled it in the cmake call.
Now build it with mingw32-make and see if it creates a dll in the src subdir and csync.exe in the client dir.

Building mirall

For mirall, it works similar. Mirall uses Qt and is C++, so again a lot of packages to install. Make again sure to have the mingw32-qt packages, for example mingw32-libqt4-devel and more.

However, there are two caveats with mirall:

• the current development state of mirall needs the latest devel version of csync which
  we just built. I tweaked the CMakefile that way that if the mirall- and csync and build*
  folders are in the same directory, the csync is found by mirall cmake in the parallel dir.
  So I do not have to install the devel version of csync in my system.
• to build mirall for windows, it must be made sure that cmake finds the mingw32 Qt tools like moc. Since there is also the Linux moc in the system, this can confuse.
  Domme pointed me to a script that sets some variables correct values to prevent mixing:
  
  cat ../docmake.sh
  # %_mingw32_qt4_platform win32-g++-cross

  export QT_BINDIR=/usr/bin
  export BIN_PRE=i686-w64-mingw32

  /usr/bin/mingw32-cmake \
-DCMAKE_BUILD_TYPE="Debug" \
-DQMAKESPEC=win32-g++-cross \
-DQT_MKSPECS_DIR:PATH=/usr/i686-w64-mingw32/sys-root/mingw/share/qt4/mkspecs \
-DQT_QT_INCLUDE_DIR=/usr/i686-w64-mingw32/sys-root/mingw/include \
-DQT_PLUGINS_DIR=/usr/i686-w64-mingw32/sys-root/mingw/lib/qt4/plugins \
-DQT_QMAKE_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-qmake \
-DQT_MOC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-moc \
-DQT_RCC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-rcc \
-DQT_UIC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-uic \
-DQT_DBUSXML2CPP_EXECUTABLE=${QT_BINDIR}/qdbusxml2cpp \
-DQT_DBUSCPP2XML_EXECUTABLE=${QT_BINDIR}/qdbuscpp2xml ../../mirall


With that setup I can build both the Linux and Windows version quite easily. There is still a lot to be solved, such as automatted packaging and such. CMake as usual is a great help.

Last week I have been on CeBIT for a day, which is a huge and busy event. We had a nice preview running there with syncing between the web client and the desktop clients on Windows and Linux, as well as on an Android device. That already worked quite ok, nevertheless I got quite some bug reports since then. And I still have to work with csync upstream to get patches in there before we’re ready for a public beta.

This weekend I will be at the Chemnitzer Linuxtage together with friends from the openSUSE project. I will show some ownCloud syncing there as well, so if you like show up at the openSUSE booth for a demo and a chat