ownCloud Planet

ownCloud
Smashbox in action
July 6, 2015

cern_logo
Last week we wrote about the history and origins of Smashbox, a versatile testing tool for ownCloud developed by CERN. Today, we will discuss how Smashbox works, where it is helpful, how to develop tests, and how it will help ownCloud in the future.

How it Works

Smashbox essentially emulates multiple users doing a number of tasks. While it can be used to emulate larger numbers of users (for performance testing), it is at its best as a functional end to end testing tool, meaning that it is meant to extensively and rigorously test the functionality of ownCloud and detect where a change might have broken something.

As an example, consider the following scenario:

  1. User one shares a folder to user two with full access rights
  2. User two re-shares the folder with user three limiting access to read only
  3. Now user three should not be able to upload in the folder

There are many dozens of variations and more complex forms of the above. It is hard and laborious to test all of them by hand, but Smashbox automates this process and can thus make sure no change breaks any of its tested scenarios.

And Smashbox can do more than just API calls to ownCloud. One of the ways it can test is by initiating multiple sync clients doing parallel and do up- and downloads and then check if the files change and are modified correctly. This makes it a very versatile tool; useful in many scenarios.
cernbox

Sharing and Smashbox

In general, testing file sharing in ownCloud is a big focus for Smashbox, and while steps have been made for ownCloud 8.1 to have very extensive testing in place, one can come up with an almost infinite number of challenging scenarios. One area which is begging for testing is the federated sharing API. The ownCloud QA team will get on that as soon as the final API is stable and published!

Smashbox for Testing Bug Fixes

Kuba pointed out that another area where Smashbox will be helpful, is bug reporting. It is often hard to reproduce bugs and users don’t always respond to requests for testing possible solutions. Developers then have to assume that the problem is solved based on their own testing or knowledge of the code. Smashbox, however, allows a developer to build a script which reproduces the bug so he/she can be confident that the bug has been fixed. Moreover, the script can then become part of the test suite, ensuring the problem doesn’t rear its ugly head again in the future.

For example, Smashbox has recently been helpful in tracking down a bug in the HTTP threading code in the Qt libraries (used by the desktop client) which could cause data loss in certain very specific circumstances.

Smashbox in Development

Smashbox is equally useful when developing new code. Upon re-factoring core infrastructure, Smashbox can test for regressions, as was its role when used to test the new high level file locking. It was also beneficial when ownCloud added end to end check summing (a client feature for now).

Smashbox and Load Testing

Smashbox can also be used to generate load on the system to test scalability or to trigger problems that only happen on a heavily loaded system. You can set up some background use, for example, you can have 200 users being active and then run the test cases and see if they still work.

Test Writing

Tests are written in Python, often by using the PyOC client library developed by Vincent Petry. Python takes care of constructing the API calls, saving the effort of manually having to write HTTP API requests.

Writing tests is lightly documented in the repository here, but the best way to write a new test would probably be to start with an existing one and building on it. Some efforts will be put in by explaining some of the existing tests better to help new test writers and the team currently working on the tests notes that Python knowledge shouldn’t be a barrier. One of our testers claims she didn’t know about it in January and was writing tests daily in March!

If you’re interested in increasing test coverage, there are some test cases in progress which are not yet developed – that might be an easy way in. Of course, if you have your own ideas, get going! Help is welcome in any way or shape.

The team is also working on merging the changes made back into the upstream project, tracking their progress in this github issue.

Future

Over the coming weeks and months, the ownCloud testing team will work on expanding the test coverage of ownCloud. An interesting side benefit of the work on Smashbox has been that it exposes functionality that is lacking in the public ownCloud API’s. Much of the functionality is in one of the multiple internal API’s used by components of ownCloud to talk to each other. To test this, an external API is needed. Smashbox not only forces the ownCloud developers to add any missing public API functionality, but to also think about what these API’s should look like and how they should relate with the internal API. It essentially provides a nice test case for each new API; providing feedback on its design and implementation.

Improvements to the ownCloud protocols and API’s are something Kuba also deeply cares about. He notes that at CERN “we are feet on the ground people” and that the usage of ownCloud and other IT services is exploding with new usage models, users, and growing expectations. The team is keen on exploring new ideas and possibilities but their foremost preoccupation is with a stable future for their services. He points out that Amazon S3 has been very successful, in no small part due to its successful ecosystem of solutions. This was made possible by a well-designed and documented protocol and he would like to see similarities between ownCloud and S3 in this regard.

Meanwhile, the CERN team is developing tests around protocol usage and implementation, which can help determine whether a problem is on the server or client side. Another new thing they are working on is ‘fault injection techniques,’ where Smashbox will intentionally misbehave (within or outside of the defined protocol) in an attempt to uncover issues with the client.

As an ownCloud app developer, a core contributor, or even just a user who is interested in helping out with testing, you should consider having a good look at Smashbox and helping out with the expansion of test coverage. After all, while running a single test once is useful in finding bugs or verifying they are fixed, writing a test case and making it part of the Smashbox test suite provides test coverage of the functionality in the future. This makes the test also helpful for catching regressions and ensuring continued functionality. Additions to the ownCloud public API should especially be accompanied with an appropriate test.

You can find the Smashbox code in github and talk to the testing team on IRC at any point in the #owncloud-testing channel or on #owncloud-dev. People most involved with testing are @jennifer, @PVince81, @nickvergessen and @morrisjobke. And of course – join the ownCloud Contributor Conference next month in Berlin to discuss Smashbox with its developers and contributors in person!

Thanks to CERN and Jakub Moscicki for their time to answer our questions

read more



ownCloud
ownCloud Release Channels
July 2, 2015

ownCloud 8.1 coming soon image roundWith millions of ownCloud users, there are a wide variety of needs and requirements. ownCloud Server 8.1 will introduce Release Channels to give our users more flexibility in choosing an ownCloud version to meet their needs. Read on to find out what they offer and how to use them.

Release Channels

Some users want the latest ownCloud in order to benefit from the new features, performance improvements, and apps. Others depend on apps, which require a specific ownCloud release, or delay upgrading to a major release for other reasons. Even other users want to help ownCloud get better– checking if a bug is fixed or testing to see if the upcoming release works for their requirements– and would like a super recent development version. To offer the “best fitting” ownCloud Release to our users, we decided to set up release channels for the ownCloud software. Users can select the best installation source based on their own demands.

You pick a release channel either from the Update app in the Administrator settings of your ownCloud instance, or by choosing a package repository of a specific stable stream. You can watch the video below or read on to hear about the release channels we offer.

These are the channels we are providing:

Production

This channel delivers the latest fully tested release of ownCloud.

That does not imply that these releases are bug free. The issues are known and can be worked around. The downside of this channel, however, is that the releases are behind in terms of features and performance.

Get It

In the updater app, this channel only delivers versions that have proven their stability in the Release channel for quite some time– one or more months after the release of a major new version.

For Linux package users, administrators can decide what release they want to be on by picking a version-specific repository. They will only receive minor bug fixes and security updates until the release is no longer maintained. This allows you to choose when to upgrade to a major release yourself, but also puts the responsibility on your shoulders of making sure that you do not run an unsupported release!

If at all possible, we suggest to upgrade sooner rather than later. Doing a test on a copy of your production environment is a better way of ensuring the new version works for you than waiting and hoping all issues are found and fixed – that way, you lose out on many of the benefits of the new release, not only in terms of features and user interface enhancements but also when it comes to performance improvements and scalability.

You can find the version-specific repositories and links to packages on the Release Channels page (which is empty until the release on July the 7th).

release channels screenshot
Choosing release channels in the updater app

Stable

The latest stable release– as it was released by the ownCloud community. This channel delivers the latest released features after we have done our internal testing as well as public Beta and RC tests and fixed all nasty new bugs found.

Get It

This is what you get by default from owncloud.org/install. If you installed ownCloud from there, you will be offered an upgrade to ownCloud 8.1 and future releases on their release date.

Beta

This channel delivers dedicated test versions such as betas and release candidates. Betas and release candidates are the versions of choice for testers in preparation for an ownCloud release.

If you want to make sure that the upcoming ownCloud release will function well in your setup, grab a beta, do a test upgrade, and run.

Get It

Join the tes tpilots to get links to the latest test versions.

obs-logo

Daily

ownCloud daily builds. It contains an automatically updated source of the current development branch. This version is not recommended for production use but is mostly helpful to reproduce bugs and follow the development for the next major version.

Get It

You can find the daily package repository and archives in the testing section of the install page.

ownCloud Server 8.1 will bring many improvements, not only to the software itself, but also around it. Besides the new Release Channels, the release will introduce improved app handling and more.

read more



ownCloud
ownCloud, CERN and Smashbox
July 2, 2015

cernbox
A few years ago, CERN choose ownCloud as a component to build an innovative sync and share service at the LHC scale. After integrating ownCloud into their architecture they needed a deeper insight into the functioning of the sync client, which led to the development of the Smashbox automated testing tool. Today ownCloud has started to adopt this versatile tool for functional testing. In this post we will discuss the origins of Smashbox with CERN IT engineer and Smashbox developer Jakub ‘Kuba’ Moscicki and how Smashbox is being used in ownCloud with some members of the ownCloud Development and QA team. Next week, you’ll hear more about how Smashbox works and how you can get involved.

Origins of Smashbox

ownCloud has been used at CERN for two years and its usage is continually increasing. Due to the challenging requirements of the researchers and the infrastructure to reliably deal with the massive amounts of data coming from the scientific devices in place at CERN, the ownCloud in use has been modified in several places. The architecture of the sync and share service at CERN, CERNBox, fuses the synchronization and web access layer provided by ownCloud with the peta-byte storage system, EOS, used to handle LHC data.

Operating a service at this scale and complexity requires advanced Quality Assurance tools in order to verify operational state of the service as well as manage service updates. This is especially true for software running on a user system as problems there leave little room for IT to help the user, unlike on the servers where backups and other recovery mechanisms are in place. Moreover, at CERN scale even obscure corner cases become common and problems require time-consuming direct interaction between IT personnel and users. And on top of that, the multi-platform nature of the client and the lack of control service providers have over the user systems further increases the number of possible versions and variations of clients interacting with the servers.
cms-event
The idea started when the CERN team had developed some scripts with test cases that exercised the syncing. When a new configuration was implemented, the scripts were run. More interesting cases were added, a name was invented and automated Smashbox testing was born.

Smashbox at CERN

Smashbox testing became a routine before any new sync clients were added to the environment. The checks are described by Kuba as ‘very mean.’ The team tries to come up with scenarios which are difficult to handle. Tests are run continuously and in varying conditions. For example, hard to reproduce issues can depend on a timing dependent problem which only happens under very heavy load on the server, requiring many hours of running the tests under a high, simulated load in order to trigger the problem often enough so they can gather data and the ownCloud Development team can track the issue down and fix it.

Smashbox and ownCloud

ownClouders first got wind of the existence of Smashbox when it was demoed at a meeting at CERN. Klaas Freitag from the client team was particularly interested, as it allowed easy testing of the API used by the client to talk to the server. Next, the CERN Smashbox repo got forked and several ownCloud developers and testers got working by writing tests and adding support for new API’s. It is currently in the public ‘smashbox’ repository on github and there is ongoing work to merge our changes back into the upstream Smashbox project. As most of the work is in the tests, rather than the framework itself, this shouldn’t be a big problem, though the entanglement of tests and framework in the current repository would probably benefit from separation.

Kuba is glad that ownCloud has picked up Smashbox. As he points out, CERN has a culture of sharing and collaborating.

Next week we will dive a little deeper into how Smashbox works, what it is being used for, and where it will go in ownCloud.

Thanks to CERN and Jakub Moscicki for their time to answer our questions

read more



ownCloud
8.1 is Nearly Here, Spread the Word
July 1, 2015

ownCloud 8.1 coming soon image round
We’re about a week away from the release of ownCloud Server 8.1! There is much that is new and improved in this release and we need your help to make sure everybody knows.

Social Media

Much of the conversation around ownCloud happens over social media. And it is your advocacy that will help others find out about the release! You can re-tweet and share the messages we put out on our Facebook, Diaspora, Twitter and Google Plus accounts and join in on the discussion about ownCloud Server 8.1 with hashtag #owncloud on Twitter, Google+, Diaspora and Facebook!

News Sites

A very important area where we could us your help is with submissions to news sites. That includes the likes of Slashdot, reddit and many others you can think of, especially those in your local language– they should have a post about the ownCloud release and you can make that happen!

Here is a nice text you can use, or translate, in order to submit the ownCloud Server 8.1 announcement to sites:

This release marks significant under the hood improvements, increasing scalability and performance of syncing and file operations while making ownCloud a better platform for developers to build upon. Security enhancements, integrated documentation links and more control in the admin panel over external storage, LDAP and encryption make ownCloud more secure and easier to use.

The link to our announcement blog will be:  http://owncloud.org/blog/owncloud-8-1-raising-the-bar-on-security-and-performance
The link starts working on Tuesday the 7th, when we publish the release. Another important link, also not yet working, will be http://owncloud.org/eight-one which is where you can find an overview of all the improvements in ownCloud 8.1!

We have also created nice banners you can put up on your blog or website to promote the upcoming release! These are the sizes available:

Example code:

<a href="http://owncloud.org/features"><img src="https://owncloud.org/wp-content/themes/owncloudorgnew/assets/img/features/ownCloud-release-460x70.png" alt="ownCloud release" width="460" height="70" /></a>

Will look like this:

ownCloud release 460x70

These banners will get updated whenever a new release is coming or has been published, so you don’t need to change anything once they are up.

The upcoming ownCloud Server 8.1 matters. The release brings many improvements (that link is going live on release day!) and we’re very proud of what is coming. Of course, in preparation of the release, you should plan on joining (or organizing) a release party to meet fellow ownClouders in real life and discuss all this awesomeness.

So get out there and spread the word!

read more



Lukas Reschke
Combining ownCloud and Google calendar for public room availability
June 24, 2015

In my coworking space we are using ownCloud calendar to keep track of the availability of our conference room which we are also renting. However, we want also to be able to show publicly the room availability without disclosing personal information to the public. Even more limiting, since we use Jimdo to host our website we can’t execute any server-side code.

Currently we have a calendar shared using a service account to all administrative users which is used to track availability. Basically looking as following:

Reservations in ownCloud calendar

Unfortunately, the default ownCloud calendar is not able to share calendar publicly, there is since a longer time a Pull Request to add this functionality but unfortunately it is unlikely that this one is going to be merged anytime soon. Also we need to be able to embed the calendar easily on other web pages.

The first solution that came to my mind is Google Calendar since it allows easily embedding calendars within web pages. However, we wanted to keep the data under our own control and also leverage our existing tools.

Luckily ownCloud is providing CalDAV access to calendars and also iCal export due to the super awesome sabre libraries that we use. Sabre offers also classes to read objects such as iCal files. This means that to remove sensitive information from an calendar and get a new iCal file only a few lines of code are required:

<?php
// Use "composer require sabre/vobject" to get the required libraries
require_once('./vendor/autoload.php');

use Sabre\VObject;

// Configure your data
$remoteHost = 'https://cloud.smartworksg.ch';
$calendarName = 'sitzungszimmer';
$username = 'service';
$password = 'XXXXXX';

// Get ownCloud calendar
$curl = curl_init($remoteHost . '/remote.php/caldav/calendars/'.$username.'/'.$calendarName.'?export');
curl_setopt($curl, CURLOPT_USERPWD, $username . ":" . $password);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_VERBOSE, true);
$ics = curl_exec($curl);
curl_close($curl);

// Parse calendar file
$calendar = VObject\Reader::read($ics);

// Replace all SUMMARY fields with a value of "Reserved"
foreach($calendar->children() as $children) {
    if($children instanceof VObject\Component\VEvent) {
        $children->SUMMARY = 'Reserved';
    }
}

// Put the resulting ICS to /var/www/public.ics
file_put_contents('/var/www/public.ics', $calendar->serialize());

This file can then get easily added to a cronjob and using Google’s calendar “Subscribe to public calendars” functionality the sanitized calendar can get shared to the public and embedded in web pages:

Reservations displayed publicly in Google Calendar

read more