From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753585AbdLMRCC (ORCPT ); Wed, 13 Dec 2017 12:02:02 -0500 Received: from mout.web.de ([212.227.17.11]:54902 "EHLO mout.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753189AbdLMRCA (ORCPT ); Wed, 13 Dec 2017 12:02:00 -0500 From: Dark Dragon Subject: Directory virtualization / Cloud filesystem To: linux-kernel@vger.kernel.org Message-ID: <6ae0eafd-0b88-823f-cb09-5c41742e7692@web.de> Date: Wed, 13 Dec 2017 17:42:47 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:OSk0C9fjD60z7Wa5zWHSXbyBPXJakzCyrg7JeHrXe7l0h71hP5M qXzGV3HnNQFC5uobeF/jQuewSKyTaj1ko9dPvYzhuiYdBUJ+hhkio2guks6GDdSip+b8g8Y yckSqzSZzMAwHo9lXgFq8gR7dvJEqxNbfbPLrTqDAlpbkoC01YRe6JERP3j6U4V6VHAYsL8 Sca3XWrFJ+s41dUklv2XA== X-UI-Out-Filterresults: notjunk:1;V01:K0:OVV6WcReUZs=:64C2BcEGG5OqpZZ7oAJpCv /FArwqm3U0SJv0VMGVibR5T1+EPC7J7Bq9zgw1TDKisLFdjFKrkd8a54Nk3IViaBSj9ozL2Za CKgTDAYLrz0/+dcTS4h8bAMWmeW+qUrFUgrkq8v8MNg8ZpDVRB4CSul/cZP4cI8yhLZHGPh/q 27ti15deNlXljj6I/f0pVu1kOr09iARDDqd4cIdHBqwhrsodvygAfgS8DtIOcA/2yPBRNk2Kh 1s1X4WIaYnyKHZh44Ci+hnnGv17nXV0fEWaCH3yGhx8bdRirk4ujpe4+ub8BWkZfeZ2zpS3rv vvh8hQDUFvfSyKic1EuhNHn8HgDgpGQmhA4DK/nwzQEBUKfXeTQUuLmHB7uy2VfbLJrqnH6Vz CmSnJvvwcx9RskHg0v9fP/UjCmEjoWlP3AaJAs5Z+TPWFboeyNcksuu5lVO5krCy0PmfFUaxp d2IAac1iaIpMSR1C+1+YGRJvpC/1PjYaghAd4nznz/Dgzp/UNMNiIfARqfN0/bdj5LLUp7BSE HkkOztqStyAxnodmF+OB44xX7gf0K64qxDDobb032id3StI10FUNCrXh98Sgy85BGJkU4hZi2 mpW4eXEqACsWN+HSxa9TA3enSK8dV8Aro2qK8jSlRdHhzIIUMcJvhDROe3apGitHo5BIIlNl6 te7ABRZLIqkqQWMvZX8VJfN8NA0245HBXoSastbFaXxL7QrX+mCFrTirgSA1y0aNg8usMJ9ER 8FfbxDWux3TWetw9A/l7uSaM33F5loMHCToRHoxptsyBwiPs6hlTxXulkZp+sfA3kY+IlBCNF QsFVnqDk1mvQvcReX5TuwN2Qw2MC3rYIHzwVQTz03WAQcxQga7YhdQvT8H9z5SfHvp/fdHu Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Is anyone working on a filesystem driver or filesystem overlay with the following characteristics: - *Kernel space* for performance reasons (no FUSE) - *All data* is stored on some server - *All metadata* is locally available - *Some file data* is locally available - *Other file data* is automatically fetched on-demand -> *All files* can be accessed from *anywhere* (explorer, console, APIs, ...) independent of their location Many projects currently work on their own (partly closed source) implementations - for performance reasons often in *kernel space* (instead of FUSE) in order to avoid context switches. In my opinion this is bad in terms of security and stability! Despite your personal opinion on whether (personal) data should be stored in the (private) cloud, it is unquestionable that there is a raising demand for these features (files are updated more often then used locally, limited fast storage in contrast to raising amounts of data, ...). My suggestion is the following: - Create a virtual filesystem (where is data stored?) or overlay filesystem (store data in underlying filesystem) in the kernel - Agree on a data retrieval protocol - File selection for intelligent caching should be performed in userspace Current projects working on this: - Seafile Drive client [1] - Git Virtual Filesystem (GVFS) [2, 3] - Dropbox Smart Sync (Project Infinite) [4, 5] - OneDrive On-Demand [6, 7] I also suggest the following news coverage about some more details: - "Microsoft and GitHub team up to take Git virtual file system to macOS, Linux" [8] -> last two paragraphs about implementation thoughts - "OneDrive done right is back, and now it works properly" [9] -> lessons learned from Microsoft's first unsuccessful implementation Links: [1]: https://blogs.seafile.com/2016/09/02/announcing-seafile-drive-client-a-new-way-to-map-seafile-storage-as-virtual-drive/ [2]: https://github.com/Microsoft/GVFS [3]: https://github.com/Microsoft/GVFS/issues/4 [4]: https://www.dropbox.com/smartsync [5]: https://blogs.dropbox.com/tech/2016/05/going-deeper-with-project-infinite/ [6]: https://support.office.com/en-us/article/Learn-about-OneDrive-Files-On-Demand-0e6860d3-d9f3-4971-b321-7092438fb38e [7]: https://blogs.office.com/en-us/2017/05/11/introducing-onedrive-files-on-demand-and-additional-features-making-it-easier-to-access-and-share-files/ [8]: https://arstechnica.com/gadgets/2017/11/microsoft-and-github-team-up-to-take-git-virtual-file-system-to-macos-linux/ [9]: https://arstechnica.com/information-technology/2017/06/onedrive-done-right-is-back-and-now-it-works-properly/