ownCloud Planet

ownCloud
ownCloud Development in the First Half of July
July 30, 2015

collaboration
Welcome to the ownCloud community update for the first half of July – covering a crazy month! We released ownCloud 8.1 and published many blogs and videos about what’s new and improved. In coding news, work on 8.2 has started with work towards moving the Files app to WebDAV, revival of the Bookmarks app and the merge of many feature branches.

If you have thoughts or contributions for a future news post, ping Jos with your input. You can read our previous report here.

ownCloud Server 8.1

The biggest news this month is the release of ownCloud Server 8.1! This release brought many new features and performance improvements. You can find a short overview of them in the announcement and on the dedicated page on owncloud.org.

We published more details about the new release in blogs like the introduction of release channels, how to upgrade to ownCloud Server 8.1, the improvements to the Documents app, and how to configure the new caching features. We also published a blog about what changed in ownCloud Server 8.1 for developers.

Simultaneously with the blogs, we’ve been releasing a series of tutorials and demos of ownCloud Server 8.1 to our YouTube channel. Check out our tutorial playlist or the list with demos!

In Other Blogs

At the beginning of the month we talked about our collaboration with CERN on testing ownCloud (part 1, part 2) and announced an update to the ownCloud android app.

Development on the Upcoming Release

With the release of ownCloud 8.1, development of 8.2 has kicked off and there is a lot going on. Below we summarize progress in some major areas.


The Files app
The Files app

Moving Files app to WebDAV

A long-time goal has seen recent progress– moving the Files app to use the WebDAV interface. Reducing the number of code paths for file management, this will make it easier to test and improve ownCloud file handling.

Currently there are two ways in which files are managed, WebDAV and private PHP AJAX end points in the Files app. By moving the Files app over to using the WebDAV API, we concentrate future efforts in one place. As the WebDAV API is used by our mobile and desktop clients as well as any other WebDAV tool, performance and stability improvements will benefit those as well.

A lot of work is required to make this work. The Files app has previews, icons, favorite and sharing information, chunked upload and much more. Some of this is already available in the WebDAV API, as the clients have had a need for it, but much will still has to be added. Some interesting challenges are related to performance, like batch file deletion or moving. Older ownCloud releases used to delete or move files one by one, creating a huge number of database, network and file system operations. Newer ownCloud releases bundle such operations, something currently not available for WebDAV.

However, fixing these issues will greatly benefit the clients as well, as they can also make use of the new capabilities to improve their own performance and feature set. Moreover, the WebDAV backend, being based on the well-known SabreDAV, has several capabilities and advantages over the custom AJAX API’s. One of those advantages is using ‘part files’ internally, which makes it easy to resume uploads or retry parts which went wrong. This also helps deal with concurrency issues where, for example, a mobile or desktop client starts to sync a file before it is completely uploaded in the web interface (example). Similar issues exist with regards to encryption.

Work on the Files app is currently tracked in progress in this issue, with a more big-picture view.

Other related work includes the ability to use webdav to serve zip files. More significantly, this also ties in with the conversations around a next generation chunking method for the client, with first a blog introducing the challenge here and a follow-up with more details to the proposal. As this work is about using the capabilities of WebDAV, the Files app will be able to use the same interface.

Right now progress on moving the Files app to WebDAV has stalled due to a compatibility issue–Internet Explorer 8 is incapable of handling the needed WebDAV commands, creating issues for the new Files app. Unless IE 8 support gets dropped or another work around implemented, introduction of the new WebDAV based Files app might have to be delayed to a later release.

The OCS logo
The OCS logo

Improving the OCS API

Complementing WebDAV, the OCS API is the other half of the external interface ownCloud offers to third party developers to access data provided by ownCloud apps. It is documented here.

There is work going on to improve the OCS API. To support Smashbox testing, an OCS API was developed for sending, receiving, accepting, and declining remote shares and OCS Share API link shares now always have a URL which fixes issue 16488. As that issue explains, this makes sure that the URL is correct (since we use the URL generator to get the route) and not depend on a client generating the correct URL. It also ensures that if the URL would change in the future the client would directly generate the new URL. It would also make it easier to write an app for URL shortening.

Another part of the effort to improve the OCS Share API is that Javascript is now used to fetch mimetype icons. This is kind of a separation of concenrs. The javascript is resposible for getting the correct icon for a given mimetype (without additional server calls) making sure ownCloud does not have to pass the icon around everywhere. This change was implemented in the Files app, bringing more responsive performance as an added advantage. In other mimetype work, users can add their own mimetypes and icons can now be changed by the theme.

Using Public API Internally

As described above, the Files app uses private PHP AJAX functionality rather than using the public OCS and WebDAV APIs to bring data from an ownCloud Server to the browser. Within the ownCloud Server, a similar situation exists where many ownCloud server apps have been developed using non-public functionality from the Core to do their work (the OC_ namespaces). Often there are public APIs (\OCP namespaces) available and this duplication is also due for cleanup.

Work on this is tracked in issue 4774 and the developers are hopeful that most instances of API duplication can be dealt with during the ownCloud 8.2 release cycle. This “dogfooding” will have similar benefits overall as those in the Files app– reducing the number of code paths allows more focused testing, bug fixing and improvement efforts.

There is quite some progress to report, for example this work which moves files_external to use public API, a pull request moving many core apps from internal OC_Log::write to OCP\Util, this PR which moves the files sharing app (mostly) to OCP\DB and this work on the files app.
php7-transparent

PHP 7 Compatibility

As you might know, the PHP community is preparing for the release of PHP 7 for the end of this year. This first major release in quite some years will bring significant performance improvements between cleanups and new features. ownCloud developers have been making some modifications to ensure that ownCloud will run on this new PHP, like this PR.

Flysystem Storage Backend Support

A new generic storage backend for flysystem adapters was merged. This allows developers to create a storage backend for any external storage supported by flysystem with minimal effort. Flysystem supports many backends already and new ones are easy to build. An example can be found in this Azure external storage backend. Now existing ownCloud external storage systems will have to be adapted to make use of this, decreasing the amount of code ownCloud has to maintain. There is already progress in this area for the FTP external storage app.

Reviving Bookmarks

Blizzz blogged about his decision to make the ownCloud bookmarks app “prosper and flourish” again. He went through the open issues and pull requests, closing many, cleaning others, merging some and fixing more. A beta release has been made and you can expect a final release with many improvements in the ownCloud app store soon.

Command Line Tool Improvements

A final improvement we want to highlight is the work on the occ command line tool. It is now possible to delete older versions of files with the occ tool. Similarly, occ can now get rid of deleted files in the trash. There was also work on a new feature to get/set configuration attributes and several improvements to configure the verbosity of the occ command.

If you want to get involved in coding on ownCloud – check out this page!
datto-ownCloud

Other ownCloud Happenings

On Planet ownCloud, a post from ownCloud.com featured a video of the Datto announcement of ownCloud partnership. This article (forbes.com) explains that Datto, a company with over 5 million customers world-wide, now ships ownCloud as part of their storage solutions. The writer points out that, in the crowded Enterprise File Sync and Share market, solutions have to provide something unique and many companies are looking for ways to store their data on-premise, benefiting from the security advantages this brings. Watch the announcement video at dattocon 2015 on owncloud.com

Note that if you’re an ownCloud contributor, you should get your blog aggregated on owncloud.org/news! Ping Jos with your RSS feed.

We hope you enjoyed this fresh report from ownCloud development!

Thanks to Vincent Petry and others for the tireless catching of embarrassing mistakes

read more



ownCloud
ownCloud Contributor Conference Program
July 29, 2015

ownCloud Contributor Conf bannerAt the ownCloud Contributor Conference ownClouders will get together for a week of discussion and coding on ownCloud. The talk program will kick off on Saturday, August the 29th, with a keynote by Angela Richter. Today, we announce our exciting program of talks and workshops, which are informing, inspiring and fun.

The Conference from a Bird’s Eye View

For designers and coders new to ownCloud contribution, the conference is the place where you write your first ownCloud app, contribute your first feature or get your first bug fix in! Basic web development skills (PHP, CSS, HTML and Javascript) are enough to get you started. Be sure to bring your laptop – we’re a hands-on community and code has to be written!

With already over 250 registrations, you are likely to find ownCloud core developers you are looking for at the conference. The existing contributors find the event a great place to discuss what they are working on, share inspiration and ideas and to get work done. The 5 days of coding are complemented with a two-day program on Saturday and Sunday with inspirational morning sessions and learning opportunities with our in-depth workshops and technical talks in the afternoon.

This event isn’t like traditional conferences – ownCloud is about making the change happen. While we have talks in the morning during the weekend, the afternoon and the rest of the week is dedicated to workshops and coding in the hacking rooms.

The Program

Most days will roughly run from 10am to 6pm, however, especially during the hacking days, the closing time tends to be more like around midnight. Saturday and Sunday we’ll open doors and registration at 9am while the introduction and first keynote will kick off at 10am with the program ending around 6pm.

Saturday

Saturday will be kicked off by an inspiring keynote from theatre director Angela Richter (interview with her here), followed by Frank, who will have some exciting announcements and great insight on where ownCloud is going.
After a short break, we’ll continue with lightning talks, including:

In the afternoon we have a number of technical talks including an introduction to Sabre/DAV by its principal author, Evert Pot, a design review session, and a tutorial on how to write ownCloud apps, among others. Remember – bring your laptop!

Saturday night we’ll have a social event with food and drinks at the beer garden close by.

See the full day-one schedule here.

Videos from the talks last year:

Sunday

Sunday continues the exciting schedule with lightning talks, covering:

The morning also features two ‘ownCloud in action’ talks, one from our atom-smashing friends at CERN and one from Sciebo, the team building a 500.000 student deployment in North Rhine-Westphalia, Germany.

In the afternoon, you can opt to be guided into contributing to ownCloud Core, the iOS app, Android app or Desktop app or dive into the Contacts app.

Find the full day-two schedule here.

Hacking Days

Following the two scheduled days, as well as the Friday preceding it, we have our hacking rooms open and expect over a hundred ownCloud contributors getting their hands dirty. The inspiration and things-learned from the two days with talks can and will be put into code. It is easy to participate – the atmosphere is entirely informal, there are no expectations, time tables or pressures; everybody is free to do and work on what they find to be most interesting. We will have easily identifiable people who are there if you have any questions, technical, social or process-wise. And of course, we provide food and drinks during these days.

Interviews we did with contributors at the conference last year

As you see, the ownCloud Contributor Conference is shaping up to be huge again! If you want to participate, be sure to register here and start booking hotel and travel! You can find our conference website with more information here.

See you in Berlin!

read more



ownCloud
Announcing the ownCloud Contributor Conference Keynote Speaker: Angela Richter
July 29, 2015

Angela Richter (photo by Elfie Semotan)
The ownCloud Contributor Conference is coming to Berlin from August 28 to September 3. The 2 day talk program will kick off on August 29 with a keynote speech given by Angela Richter, house director at the Cologne National Theater and director of the interactive project “Supernerds,” on the topic of digital surveillance, whistle blowing and digital dissidents. She has worked with various digital activists and hactivists such as Julian Assange, Daniel Ellsberg, Edward Snowden, Thomas Drake and many others, with the goal of bringing their story to a wider audience. Read on for a short interview with her.

Who is Angela Richter?

We asked Angela to introduce herself.

I am currently one of the four house directors at the Cologne National Theatre “Schauspiel Köln.” In recent years I have worked with such digital activists as WikiLeaks founder Julian Assange, and in 2012 premiered my piece “Assassinate Assange,” which was shown in Berlin, Hamburg, Cologne and Vienna. Since then, I not only visited regularly with Julian Assange in London, but have engaged in panel discussions, conferences and in various print media including Der Spiegel, Monopoly, Interview Magazine, and Revue Magazine for the Next Society for internet activists and hacktivists. During the 2014/15 season, I wrote and directed the interactive project “Supernerds” (info about Supernerds in English, German review) on the topic of digital mass surveillance, whistleblowing and digital dissidents in collaboration with national television and radio WDR, Schauspiel Köln and the Producers Gebrueder Beetz, including an “Internet Sudden life Game.” The audiences in the theatre, on TV and internet were part of the story; they experienced examples of hacking and surveillance on their smart phones and laptops. Supernerds is seen as a unique example of Transmedia because it allowed audiences to explore the story in several medias such as internet, sudden life gaming, radio, TV and live on stage.

The play was based on interviews that I conducted in the last three years, with whistleblowers, human rights lawyers, hackers and internet activists, among them Julian Assange (Wikileaks), Daniel Ellsberg (Pentagon Papers), and the NSA Whistleblowers Bill Binney, Thomas Drake and Jessely Radack. I also visited Edward Snowden in Moscow in February 2015 (German interview with him here). I published a selection of the interviews in the Book “Supernerds – Conversations with Heroes” with Alexander Verlag Berlin, both in German and English.

What do you see as the biggest problem with regards to privacy in our society? And what should we do about it?

Not enough awareness in the mainstream population. Politicians rather smother it up and downplay it. Mainstream media doesn’t shed light on it.

We need both technical and social solutions. First enlightenment to invigorate awareness about it. And also tech solutions; encouragement to make people less dependent on big Internet companies, who incapacitate people and hold them in dependency. Encouragement of free software, open source etc.

Your work and what you did with Supernerds (and other initiatives) gives you a ‘different’ perspective. How do you see this influence the debate and why does it matter?

As I am an artist and not a techie, I think I can communicate these issues without being intimidating. Through a project like Supernerds I can make people understand it easily and in a playful way. If I can get it as an artist, everybody can. Also the difference between knowing and really understanding how sensitive we are, in terms of surveillance, is important

At the ownCloud Contributor Conference, Angela will share some of her experiences and insights in both the issues she discusses in her work as well as in the work itself.

You can find the full ownCloud conference program here.

Register now for the conference and make sure you don’t miss her!

read more



ownCloud
ownCloud development in April, May and June
July 23, 2015

work in progress
The busy time leading up to the ownCloud release has made it hard to keep up with development. As the value of outdated reports is minimal, this report will merely summarize work done over the last three months, after which we’ll put renewed effort in keeping up by re-instating weekly reports. These will have a reduced area of coverage to make regular updates feasible.

This report, therefore, covers May and June, focusing on what took place in the core repository of ownCloud.

During April, May and June developers were mostly concerned with getting ready for the ownCloud 8.1 release. Since our report on March there have been feature additions, but finishing functionality merged during March has been the focus.

Development

In the three months we are covering here there have been over 560 pull requests that were merged in core alone – with a total of about 1215. As much work outside of the core takes place by direct commit rather than through pull request. It isn’t farfetched to conclude that core might be central to ownCloud, but at least half the work that benefits ownCloud users takes place outside of this repository, in the many apps developed by the ownCloud community. And that is not even covering those apps developed outside of the ownCloud project!

Check out the feature overview of ownCloud 8.1
Check out the feature overview of ownCloud 8.1

New Features

Not all major new ownCloud 8.1 features were in before April 1st. These were merged between April and the end of June:

Performance work

As a taste of some of the performance-related PR’s, this introduces a stat cache for WebDAV and this one cache’s language choice, while this improvement to the preview class helps bring preview generation time up from minutes down to seconds. To allow storage to deal with copy and move commands more efficiently, this PR implements logic to let backends handle these operations and this PR makes it possible to directly access storage when working over WebDAV, saving lots of database accesses.

There has been a lot more of these, one of the reasons why ownCloud 8.1 is significantly faster than previous releases.
cernbox

Smashbox Testing

Another area where a lot of effort was put in over the covered time period was to set up Smashbox for testing ownCloud. Smashbox is an automated testing tool developed by our Atom-smashing friends at CERN. We covered this in two blogs introducing Smashbox and describing how it is put to use.

Locking Work

ownCloud 8.1 has seen quite some work on file locking, like the addition of a file lock during download. This is part of an effort to make sure ownCloud cannot, even in the weirdest of circumstances, overwrite or lose files (see the many items tagged with feature: locking in github). ownCloud already does file locking to ensure two users can’t access the same file at the same time, but this takes place at the file system level. This will stop one user from overwriting a file being written by another user – but in between separate file operations, such as each individual file being uploaded to a folder, other operations can still take place. If, for example, one user uploads a big file to a folder on external storage which gets renamed by another user at the same time, strange things could happen. The new, high-level locking will ensure that the user who started his/her endeavor last will receive an error message, while the first user will finish successfully. In case of the sync client, it will automatically retry later.

ownCloud 8.1 introduces high level file locking as an experimental feature which has to be turned on in config.php. As it is only relevant to very large installations, while being very invasive and recently developed, it was decided to keep it off by default.

More

making the download preview for text look better
making the download preview for text look better

If you want to get involved in coding on ownCloud – check out this page!

Other ownCloud Happenings

We published a lot of blogs on Planet ownCloud during the last three months:

Please note that if you’re an ownCloud contributor, you should get your blog aggregated on owncloud.org/news! Ping Jos with your RSS feed.

We hope you enjoyed this development overview! Stay tuned for an overview of what happened in the first weeks of July.

read more



Morris Jobke
Verbose ownCloud upgrade
July 23, 2015

ownCloud offers the ability to upgrade an owncloud instance via command line. In the past this was a very quiet command and only listed the very rough steps that were executed. On instances with more user data this results in long times without any visible progress.

$ ./occ upgrade # 7.0 -> 8.0
Turned on maintenance mode
Checked database schema update
Checked database schema update for apps
Updated database
Disabled 3rd-party app: gallery
Updated <files_sharing> to 0.6.2
Turned off maintenance mode
Update successful

This is totally fine if everything works as it should. But there were a few drawbacks with this output. When an app gets updated this is only printed once it is done. It would be more helpful to see which app gets updated next, because then you exactly know where to look for problems.

Beside that there are repair steps executed before and after the DB update. It would be nice to also list these steps and know a bit more about what happens during and upgrade.

These two enhancements landed in ownCloud 8.1.1 and the upcoming 8.2.0 when you specify the command line option -v which is widely used in other command line tools to increase verbosity too.

$ ./occ upgrade -v # 8.0 -> 8.1
Turned on maintenance mode
Repair step: Repair MySQL database engine
Repair info: Not a mysql database -> nothing to do
Repair step: Repair MySQL collation
Repair info: Not a mysql database -> nothing to no
Repair step: Repair SQLite autoincrement
Repair step: Repair duplicate entries in oc_lucene_status
Repair info: lucene_status table does not exist -> nothing to do
Repair step: Repair config
Checked database schema update
Checked database schema update for apps
Updated database
Repair step: Repair mime types
Repair step: Repair legacy storages
Repair step: Repair config
Repair step: Clear asset cache after upgrade
Repair info: Asset pipeline disabled -> nothing to do
Repair step: Generate ETags for file where no ETag is present.
Repair info: ETags have been fixed for 0 files/folders.
Repair step: Clean tags and favorites
Repair info: 0 tags for delete files have been removed.
Repair info: 0 tag entries for deleted tags have been removed.
Repair info: 0 tags with no entries have been removed.
Repair step: Drop old database tables
Repair info: Table locks has been deleted
Repair step: Drop old background jobs
Repair step: Repair outdated OCS IDs
Update successful
Turned off maintenance mode

Because it is more fine granular you also know more precise what the upgrade process is actually doing.

Upcoming enhancements

In the next version of ownCloud we also will add a timestamp to each line of output while the -v option is specified. Then the admin knows exactly at which time a step happened and can give rough estimates which steps took how long.

$ ./occ upgrade -v # 8.1 -> 8.2
2015-07-23T07:32:25+00:00 Turned on maintenance mode
2015-07-23T07:32:25+00:00 Repair step: Repair MySQL database engine
2015-07-23T07:32:25+00:00 Repair info: Not a mysql database -> nothing to do
2015-07-23T07:32:25+00:00 Repair step: Repair MySQL collation
2015-07-23T07:32:25+00:00 Repair info: Not a mysql database -> nothing to no
2015-07-23T07:32:25+00:00 Repair step: Repair SQLite autoincrement
2015-07-23T07:32:26+00:00 Repair step: Repair duplicate entries in oc_lucene_status
2015-07-23T07:32:26+00:00 Repair info: lucene_status table does not exist -> nothing to do
2015-07-23T07:32:26+00:00 Repair step: Repair config
2015-07-23T07:32:26+00:00 Checked database schema update
2015-07-23T07:32:26+00:00 Checked database schema update for apps
2015-07-23T07:32:26+00:00 Updated database
2015-07-23T07:32:26+00:00 Repair step: Repair mime types
2015-07-23T07:32:26+00:00 Repair step: Repair legacy storages
2015-07-23T07:32:26+00:00 Repair step: Repair config
2015-07-23T07:32:26+00:00 Repair step: Clear asset cache after upgrade
2015-07-23T07:32:26+00:00 Repair info: Asset pipeline disabled -> nothing to do
2015-07-23T07:32:26+00:00 Repair step: Generate ETags for file where no ETag is present.
2015-07-23T07:32:26+00:00 Repair info: ETags have been fixed for 0 files/folders.
2015-07-23T07:32:26+00:00 Repair step: Clean tags and favorites
2015-07-23T07:32:26+00:00 Repair info: 0 tags for delete files have been removed.
2015-07-23T07:32:26+00:00 Repair info: 0 tag entries for deleted tags have been removed.
2015-07-23T07:32:26+00:00 Repair info: 0 tags with no entries have been removed.
2015-07-23T07:32:26+00:00 Repair step: Drop old database tables
2015-07-23T07:32:26+00:00 Repair step: Drop old background jobs
2015-07-23T07:32:26+00:00 Repair step: Remove getetag entries in properties table
2015-07-23T07:32:26+00:00 Repair info: Removed 0 unneeded "{DAV:}getetag" entries from properties table.
2015-07-23T07:32:26+00:00 Update successful
2015-07-23T07:32:26+00:00 Turned off maintenance mode

read more