Kolab.org Groupware Solution

I’m working on enabling continuous deployment with help of Nulecule, so that I can have my developer’s edges as well as a series of central environments entertain deployment based on the same triggers. For those of you outside the inner circle ... [More] , Nulecule is supposed to abstract deployment to different cloud orchestration providers, whether vanilla Docker, something assisted by Kubernetes, or something as fancy as OpenShift. In the Kolab Groupware world, orchestration of micro-services is no new challenge. We operate many, many environments that deploy “micro-services” based on (fat) VMs, but the way this works based on (thin) containers is different — there’s no running Puppet inside a container, for instance, or if there is, you’re doing it wrong. So, aside from the fact we know what our micro-services are across the application suite, the trick is in the orchestration. I have some 25-30 micro-services that require access to a service account’s credentials, because we do not normally allow anonymous binds against LDAP (any exploit anywhere would open you up to reconnaissance otherwise). Currently, setting one container to hold and configure those credentials in a canonical authentication and authorization database and distributing those credentials to the other 24-29 containers doesn’t seem possible without the user or administrator installing Kolab using containers specifying the credentials 25-30 times over. On a similar note, it is currently not possible to specify what volumes for a Docker container are actually supposed to be persistent storage. The point to take from this is to appreciate that Kolab is often at the forefront of technologies that have little to do with bouncing back and forth some emails among people that just so happen to share a calendar — in this case specifically, we’re abusing our use-case’s requirements with regards to orchestration to direct the conversation about a unified format to describe such requirements.   [Less]

Recently I implemented a couple of improvements that are related with files storage. It’s better access rights support and storage integration with more components. Add attachments “from could” It was already possible to attach files from file ... [More] storage in email compose page. Now you can also get files “from cloud” in event and task creation/editing forms. Figure 1. “From cloud” button in event dialog. Read-only folders As for now the user interface where folders from file storage were used wasn’t aware of folder access rights. Now it’s changed. Chwala API provides information about access rights. So, the folders list displays a locker icon for read-only folders. Also, when you use “Save to cloud” feature, the list will contain a list of writable folders only. Additionally, any write operations (like file upload, delete or move) are prevented by the UI, which means e.g. some buttons are inactive in read-only folders. Figure 1. “From cloud” button in event dialog.   [Less]

Kolab Groupware is a collaboration suite establishing the integration of various applications you know already; Most prominently, these include 389 Directory Server, Postfix, Cyrus IMAP and Roundcube. Together, these applications would comprise a ... [More] simple mail system that, in terms of functionality, would fall short of “groupware” and “collaboration” functionality. Side-note: Cyrus IMAP has added CalDAV, CardDAV and WebDAV capabilities, encroaching on the territory Kolab otherwise occupied, rather exclusively, for as far as the world of Free Software is concerned. As such, Cyrus IMAP provides more of the groupware functionality than I initially stated, but does not provide ActiveSync capabilities, a web-mail client interface, Resource Management, and many other facilities included with Kolab Groupware. Inversely, however, Kolab is going to need to show to be the most adaptable, and applaud and welcome and embrace these developments in Cyrus IMAP, rather than attempt to compete with it somehow. This blog post is about Single Sign-On and second factor authentication, however, and where and how these fit in with Kolab. First, we need to clarify the terminology, because SSO and 2-factor authentication both mean different things to different people. Single Sign-On Single Sign-On is the functionality in which a complete infrastructure of services provided allows you to authenticate precisely once, and only ever once, from which point on forward you are trusted to have identified yourself — contrary to where you can use the same credentials over and over against different services, but you have to fill out and submit your credentials again and again to create valid sessions. The ability to use the same credentials everywhere is usually achieved by making individual applications use the same authentication and authorization database in very much the same way — usually LDAP in some form or the other, or a SQL database. In principle your credentials fly over the wire time and time again in exactly the same way they would otherwise, they just happen to be the same credentials every time. That is not to say that in a Single Sign-On scenario, no credentials go over the wire. They just so happen to not be your username and password, but separate ones issued to you, and negotiated the level of trust for between multiple parties — you, the issuer and the service. The clearest, cleanest and best example of this is Kerberos. Two-Factor Authentication Two-factor authentication — or multi-factor authentication — involves the functionality to supplement what you know (your password) with something you have. This translates to a token of sorts, usually presented in a physical form or encasing (an application’s configuration on a smart phone), that you are required to have on you and is expected to never change ownership. This can be a YubiKey, a smartcard, a phone, and such. Since these devices are eligible to contain a form of processing power, most common implementations of the second factor today are one-time passwords — supplemental tokens that are valid precisely one time, or for a limited window in time — without involving space it is understood this is sufficient. In the realm of Kolab especially, your username is often widely known — after all, an email address is rather limited in functionality unless your mailbox is to remain empty. More broadly speaking, your username tends to be known regardless — Twitter handles, Google email addresses, etc. Back to Kolab As I’ve mentioned before, Kolab has multiple access points — IMAP, SMTP, LDAP, the various web applications such as Roundcube, ActiveSync, CalDAV, CardDAV and WebDAV. The difficulty in applying Single Sign-On and Two-Factor Authentication needs to be viewed in the context of some of these separate applications being required to authenticate to other applications as you. How, though, does a web-mail client interface authenticate as you when it needs access to IMAP, LDAP or SMTP? Proxy Authorization? The first option is to consider the fact that IMAP, LDAP and SMTP have a concept of “proxy authorization”, where a set of credentials other than your account’s is used to authenticate, and the connection is subsequently authorized as if it had been your account logging in to begin with. More specifically, IMAP and SMTP provide this functionality as part of their use of SASL, and LDAP provides these capabilities separately. However, this means these credentials would need to be available to the client application. Using these credentials with additional privileges (proxy authorization) over the user’s credentials constitutes privilege escalation, hopefully followed by de-escalation, and is therefore generally considered bad practice, especially in the context of services exposed to the Internet. Central Authentication Service? Another option is the Central Authentication Service (CAS). With such a service, you typically authenticate via a web application, which submits your credentials to CAS for verification. The web application is then issued tokens representing the validity of your session. When the web application in question needs to authenticate to another service as you, it submits the temporary credentials associated with the specific session, and therefore allows the third party application to validate the tokens in question as being a part of an existing session (again via CAS), operating under conditions to be expected — i.e. Roundcube connecting to Cyrus IMAP. Contrary to popular belief, this isn’t necessarily limited to just web services doing so, and can perfectly well be applied to non-web service endpoints as well. However, a full desktop client to IMAP, such as Kontact, would not necessarily understand the need to consult a web service for obtaining tokens then valid against IMAP or SMTP. IMAP nor SMTP (or, actually, SASL) itself does not provide an authentication mechanism through which a full desktop client can be told to facilitate such type of external authentication. OTP for Two-Factor Authentication Therein lies the problem with OTP as well. There is currently no mechanism in which IMAP or SMTP (again, actually SASL) can be told to authenticate against a website first. While it can be required to issue an OTP, no SASL server-side implementation I’m aware of allows either of these to occur through a centralized authentication service. Feel free to leave your recommendations in the comments. “But, …”, you might say, “… does not Google implement an XOAUTH2 mechanism?” or “Is there not RFC 7628?” — to which the answer is “Yes, but …”. Cyrus SASL does not currently support OAUTHBEARER nor XOAUTH2 mechanisms. Alternatives like gSASL do not do so either. No full desktop client software that I’m aware of supports it — feel free to leave your recommendations in the comments below. Kolab does not currently support it, in that it is not an OAuth or OAuth2 provider. Several other mechanisms achieving a similar relaying of the authentication process to occur outside of the IMAP or SMTP connection — SAML20 and OPENID20 come to mind, both of which are implemented in GNU’s SASL. For what it’s worth, Kolab is not a SAML nor OpenID provider currently, either. I could, if I wanted, break these down in to work units — enhancement tickets if you will. I have, in fact, opted to include an OAuth provider in my new PACK project for this reason among others. Either, though, is slightly beyond the scope of this post (albeit not this blog). Identity Management A third option is viable only for fully integrated Identity Management solutions, where FreeIPA comes to mind. With FreeIPA, we may consider the Kerberos ticket issued valid, optionally so only after a valid second factor has also been submitted to obtain the ticket. Relaying the authorization is a bit of a pain, but we’ve managed in a Proof-of-Concept implementation for which I did the work, and the patches against Roundcube and the Kolab-specific authentication plugins have been accepted. The deployment scenario under which this is considered a viable option however is rather limiting. You do not typically deploy a Kerberos environment on the Internet in the first place (while work is ongoing to make that a more reasonable approach), but then still, a single client operating system (on to which the user logs in and applications get launched) is also typically associated with a sole provider environment. Long story short, I associate this typically with a single environment within a single organization, rather than a hosted or even colocated Kolab Groupware deploment. Conclusion While not one particular conclusion set in stone, several options result in various combinations of units of work. We could protect access using OTP and subsequently allow only the web client to be used by those that feel all access must be protected with OTP. We could use CAS to further reduce the number of points where copies of credentials need to be saved off as those points require access to other services as you, while protecting privilege escalation vector-based attack surfaces. We could use an OAuth2 provider to negotiate the authorizations for the various applications that Kolab includes. Comments and ideas welcome, as I feel we’re (I am) breaking ground on new territory.   [Less]

I now have a bit more information on what it is I will be achieving, and I wanted to share the roadmap and horizon of the project I’m undertaking. The software development project is called PACK — a portal or panel for administration of Kolab ... [More] Groupware. For our current most important deployment, Kolab Now, this involves customers and accounting features. This would include, for example, a Web Administration Panel and a Customer Control Panel. For the purpose of this intermezzo, we’ll simply state that some parts of the overall application suite address different needs for different audiences in different deployments. To illustrate, an on-premises installation likely has little need for third parties to be able to autonomously register themselves as customers. However, depending on the internal accounting practices of such organization, a department may need to fork over part of its budget to the IT department for providing them with collaboration services. That being said, I’m in for quite a lengthy and complex project. On the horizon may be a rather complete encapsulation of quite a few too many things to include in a first few milestones. As I’ve mentioned before, I’m following my own learning curve. There’s some troublesome areas in Flask and its extensions, where modelling of the applications to include multiple features based on multiple extensions is tricky. One such example is including localization (l10n), internationalization (i18n), themes, assets and caching in to one application. Furthermore, the extent to which we wish to include features makes it very difficult to come up with a data model and database design that encapsulates everything. So, where do we get started? Well, the use-cases that are top of mind are limited to those we have for Kolab Now and Kolab Enterprise. This means we let individual users register or sign up, and subscribe to entitlements on product offerings. This includes a nice little range of offerings, such as; Knowledge base articles, some of which may be open to the public, some of which may be open for logged in users with no further entitlements on any product, some of which may be available only to users with particular entitlements. These articles form an extra on top of any amount of entitlements. Subscription Management, for those on-premises installations of Kolab Groupware with long-term support and add-on software channel entitlements. Individual User Accounts, which are the equivalent of the current singular @kolabnow.com groupware account. Group Manager Accounts, where you register with a domain name you own, and get to create however many accounts you wish. Colocated Kolab deployments, where you register with a domain name you own, and in return you get your own, managed installation of Kolab Groupware — with the intention to allow for a lot more customization and integration compared to the two former hosted account types. Compared to how you sign up to a hosted Kolab account today, these individual entitlements should be separate from your base account. With this I mean that you should be able to create an account that holds no entitlements and requires no payment, and then subscribe to entitlements some of which may require payment right-away or at some point in the future. This parts with the current registration process, where you subscribe directly to one entitlement and one entitlement only, and the entitlement you register for will continue to be a different account for each entitlement. To illustrate why we part with that process, we only have to look at a family situation. A couple of parents may wish to use Kolab Groupware, but would logically like to avoid two different administration accounts being issued two different invoices for the two different accounts. Another reason to part with the current process of registration directly to individual entitlements is to encourage users to try different Kolab offerings before they commit to buying either of the forms that Kolab comes in. This immediately draws the need to be able to clearly distinguish between an account on the access portal and the accounts registered once access to the portal is obtained. The  easiest way to let people “sign up” and explore is to allow them to identify themselves with one of their existing identities — a Twitter, Facebook or Google account is what we have currently included. However, I have since learned that that is not secure enough. I have added on top of that four second factor authentication mechanisms: Passwords, TOTP, HOTP and TAN via SMS. I have since also added registration with email and password, where we would not require you to confirm ownership of or access to the mailbox of the email address you enter before we consider it a valid login name to the portal — remember we are not issuing any entitlements to this initial registration. We intend to then let you restrict access to your account further by at least adding the second factor, and once you subscribe to product entitlements, switch over to that new account as the canonical account for access to the portal. This scenario would be applicable should you opt for, say, an @kolabnow.com individual account, to which you could switch over the primary login, and void the third-party access. For support subscription entitlements however, this would not apply equally as much — since you would not have another account with Kolab Now to switch over to. Suffice it to say I have learned too much to continue the Flask Mega-Tutorial in the same way I originally envisioned I would. I’m going to have to scratch my head for a little while, to figure out what it is I want, need and require, and perhaps start over with the mega tutorial taking in to account the various things I’m going to be doing, and the seemingly iterative process of restructuring the example code layouts.   [Less]

There has been a new release of Roundcube: https://roundcube.net/news/2015/09/14/updates-1.1.3-and-1.0.7-released/ I noticed that because Epel already has version 1.1.3, but Kolab 3.4 Updates still has 1.1.2. Now there is an installation conflict ... [More] , because yum wants to use the epel version, but that leads to other conflicts. A temporary solution is to exclude all roundcubemail* packages in the epel repo file: sed -i "s#enabled=1#enabled=1\nexclude=roundcubemail*#g" /etc/yum.repos.d/epel.repo The proper solution is to upgrade the roundcubemail package for Kolab 3.4 Updates on OBS. I was slightly confused which tarball to use, and Daniel Hoffend aka dhoffend helped me out: Go to https://github.com/roundcube/roundcubemail/releases Get the commit id for release 1.1.3: 357cd5103d1c27f8416ef316c4a4c31588db45b8 git clone https://github.com/roundcube/roundcubemail cd roundcubemail git checkout -b newrelease 357cd5103d1c27f8416ef316c4a4c31588db45b8 git archive --prefix=roundcubemail-1.1.3/ HEAD | gzip -c > ../roundcubemail-1.1.3.tar.gz To test the new package, download this repo file: yum install yum-utils yum-config-manager --add-repo https://obs.kolabsys.com/repositories/home:/tpokorra:/branches:/Kolab:/3.4:/Updates/CentOS_7/home:tpokorra:branches:Kolab:3.4:Updates.repo yum update The new updated package will hopefully arrive in Kolab 3.4 Updates within the next days.   [Less]

In this article I described how we implemented client-side encryption in Roundcube using Mailvelope. There’s another approach for encryption, it is the Enigma plugin. It implements all the functionality using server-side GNUPG software. So, the big ... [More] difference in these is that: Mailvelope keeps your keys in the browser, Enigma stores them on the server. In the current state Enigma however, has a lot more features. Installation and settings To use Enigma just enable it as any other plugin. Then in Preferences > Settings > Encryption you’ll see a set of options that will give you possibility to enable/disable encryption-related features. NOTE: As keys are stored on the server, make sure the directory used as a storage has proper permissions, and it’s good to move it somewhere out of the location accessible from the web (even if secured by .htaccess rules). Figure 1. Encryption preferences section. Keys management To manage your keys goto Settings > PGP Keys. There you can generate a new key pair or import keys. See the following screenshots for more details. Figure 2. Key generation form. Figure 3. Key information frame. Composing messages In message compose screen a new toolbar button is added with popup where you can decide if the message have to be signed and/or encrypted. The behaviour and the icon is slightly different than the one used for Mailvelope functionality. Also, note that we did not change the compose screen in any way, so all standard features like responses and spellchecking actually work. Figure 4. Encryption options in compose. Summary You can find the Enigma plugin code in Roundcube 1.0 and 1.1, but only the version in Roundcube 1.2 (current git-master) is usable. I put a lot of work into this plugin and I hope there will be users that will use it. It depends on you if that solution will be extended with S/MIME or other features in future versions. Current state is described in the plugin README file .   [Less]

The most valuable feature of incoming Roundcube 1.2 release is PGP encryption support. There are two independent solutions for this, Enigma plugin and Mailvelope. In this article I’ll describe what we achived with Mailvelope. The integration code was ... [More] mostly written by Thomas Brüderli and only slightly improved/fixed by me. It looks like Mailvelope is the best (if not only) solution for encryption in a web browser. It’s based on OpenPGP.js that is an implementation of PGP encryption in JavaScript. Mailvelope is distributed as Chrome and Firefox extensions. It supports some email services like GMail, it also provides an API for programmers. And this is the way we decided to integrate it with Roundcube. Mailvelope installation For more info goto Mailvelope documentation. To have it working with Roundcube you have to install the extension in your browser, then goto your Roundcube webmail and using Mailvelope “Add” button add your page to list of mail providers. One last required step is to enable API use on the provider edit page. Compose an encrypted message If Roundcube detects enabled Mailvelope, new button will appear in compose toolbar. It may be disabled in HTML mode, so switch to plain text. If you click it Mailvelope frame will appear. There you can write your message and add attachments. As you notice on the screenshot some features are disabled. Unfortunately, at the moment we can do much more with the Mailvelope textarea. Note: to send an encrypted message you first have to import/generate a private key in Mailvelope settings. Figure 1. Message compose screen with enabled encryption frame. When you try to send a mail to an address for which no public key was found in the Mailvelope database (keyring), you will be provided with possibility to search public key servers and import the keys. Figure 2. Key search result in compose. Preview an encrypted message Also in a message preview Mailvelope will add its frame containing decrypted text and attachments. You’ll be prompted for a key passphrase when needed. Figure 3. Encrypted message preview. Summary Unfortunately this is way from being complete. The Mailvelope API is very limited at the moment. It does not support signing and signature verification. Access to the encryption frame is limited. There are also some bugs. Currently you can only send and receive simple encrypted messages (attachments are supported). You can track progress and read about the issues in this ticket.   [Less]

Welcome back to my mega-tutorial on Flask. If you’re following along with Part I and Part II, you should already have a minimal Flask application that doesn’t really do anything meaningful. While I had said before, this part would be all about ... [More] testing, you may think “Why? It doesn’t do anything!”. That’s correct. It’s time to appreciate the requirements on the application. You may or may not have an extensive comprehension of some of the things it may need to do — I did when I got started, but what precisely the application needed to do was not yet clear. What the application is going to need to be able to do, and how the application will go about achieving exactly that is subject to specification. Clear specification of the requirements, as well as clear specification of how to achieve those targets. This part revolves around testing, but I know I’m going to need to be more specific — it is about unit testing, and arguably also functional testing, with help of fixtures. We’ll be using Python’s unittest in combination with Flask-Fixtures. We’ll also use the fixtures to provide your application with some content, so it is not as dull outside of testing — the tests first wipe the database, then populate the database, but tear down and drop all tables from the database when they are done. First things first, I propose you put your tests in the directory named, well, err… tests/. A simple naming convention to ensure tests are ordered (correctly), is to name the individual files in a pattern test_$x_topic.py, such as for example tests/test_000_schema.py. This is the first give-away. Test your schema to ensure it function as you expect it to. This means when you expect an ON DELETE CASCADE to function such that the associated items are removed when you remove the referred item, you test this. So let’s start a database model. Again, I cannot emphasize enough how important it is to get your requirements sorted. In our example, we’re going to have items and one owner per item. The following is an example of ppppp/db/__init__.py. Note that I would normally put different classes in to different files inside of a model/ subdirectory, … useful especially when database models grow larger. This enables us to import the database and model in ./manage.py so let’s do that: If you were to run ./manage.py now, you’ll find tracebacks about the database not yet having been created, which is correct: $ python -c 'from ppppp.db import db; db.create_all()'This should resolve that problem. However your database is empty. This is where fixtures come in. Create tests/fixtures/owners.json, and put in it: Create tests/fixtures/items.json, and put in it: Now, we’re going to switch over ./manage.py to use the Flask-Script extension: And we can now run: ./manage.py load_fixtures. This will drop the database tables and re-create them, and add our fixtures. You should now also use ./manage.py runserver to run the actual webserver, and when you do, you should see: { "Clock": "Jane", "Watch": "John" }Now we can start testing. Again, a nice little naming convention to ensure tests are executed in order may be to name them test_$x_$subject, as follows: Let’s read back as to what we’re verifying; In test_001_owners(), we verify the tests/fixtures/owners.json file was loaded successfully, and completely. Boring, but necessary. In test_002_items(), we verify the tests/fixtures/items.json file was loaded successfully and completely. Boring, but even more necessary than the test for tests/fixtures/owners.json, because of the foreign key restrictions. In test_003_owner_delete_cascade() however, we test the real deal: Is the item deleted when the user is? It is important to realize that just writing this one test would fail to pass the mark, because an owner Jane could simply have no items, because the schema, fixtures or even the tests could have been (written) wrong. Running the tests would result in: $ venv/bin/nosetests -v tests test_001_owners (test_000_schema.TestSchema) ... ok test_002_items (test_000_schema.TestSchema) ... ok test_003_owner_delete_cascade (test_000_schema.TestSchema) ... ok ---------------------------------------------------------------------- Ran 3 tests in 2.018s OKHence verifying we have specified the ON DELETE CASCADE part of the schema correctly. At this point, I would suggest inserting more tests to create a duplicate entry John or Jane, set a unique constraint for Owner.name, etc. However, we cannot test the web application in full. Adding a test such as: Will fail with a 404 Not Found error. Why? Because the app we are using does not use the routes we have in ./manage.py. Here’s where that ppppp/web/ directory comes in. In effect, ./manage.py only needs a very minimal configuration of the application, so that it can load fixtures and do other such stuff, but more importantly, run what is otherwise app. It could import that from some location so that other stuff can also import it from that location. We’re going to be using ppppp/web/webapp.py for this: And we make it so it can be imported sanely (i.e. without having to expose everything), through ppppp/web/__init__.py: Now we can dumb down ./manage.py quite a bit: The tests we can now write include interaction with the application, without having to copy routes and other such application logic across all tests.   [Less]

This is a nice feature on desktop, so it is in desktop-like web applications. Average user likes this feature and uses it. That said, we have in Roundcube some drag’n’drop capabilities, it is: Dragging messages to folders (in messages list view) – ... [More] copy/move action. Dragging folders to folders (in Preferences > Folders) – move action. Dragging contacts to groups/sources (in Addressbook) – copy/move/assign to group action. Dropping files from desktop to compose attachments. As of yesterday it is also possible to drag’n’drop attachments from mail preview to compose window. If you use Kolab, you can actually also drop files from desktop into Calendar, Tasks and Files. You can also re-arange messages list columns using drag’n’drop technique. These should work in most of web browsers (their recent versions). The question is: can we have more of this? E.g. wouldn’t be nice to drag attachments or messages from browser to desktop? Or messages to compose attachments? Well, it would, but it’s not so simple… I recently investigated what recent web standards and browsers provide in this regard. This does not look good. Standard way of drag’n’drop is DataTransfer and some events defined events, plus HTML attribute ‘draggable’. Unfortunately there’s no standard way of dropping a file to the desktop. So, what options do we have: Chrome browser supports its own DownloadURL parameter of DataTransfer, but even Chromium does not use it. In Firefox you can drag a link which on Windows will create a file link, so not what we want, but under Linux (KDE) it actually can download a real file. Unfortunately, this does work only with public files. It does not work with session-based apps (as Roundcube). We’d need to implement something like one-time-public URIs to attachments. I didn’t find any information about other browsers. So, as you see not much we can do today. There’s another issue, this will anyway do not work with Roundcube widgets implementing its own “internal” drag’n’drop, e.g. messages list. Also, there’s no standard to drag many resources at a time so we cannot replace our “internal” implementation.   [Less]

JMAP is a JSON-based API for synchronizing a mail client with a mail server. As you may be aware, Kolab is a lot more than just a mail server, and in our endeavours to bring you the next-generation experience for collaboration, JMAP is a very ... [More] interesting candidate for our web client — you may know it as Roundcube Next. As with so many things on the leading edge of software development, figuring out how to run an environment suitable for development can be cumbersome. The particular case in point is that no IMAP server currently supports the JMAP protocol natively — albeit work is ongoing to change that. Our friends over at FastMail have developed a JMAP proxy in Perl, currently deemed the most complete implementation, but it is a bit of a hassle to get it up and running. So, in order to save us significant chunks of time and not run in as many circles as much, I have created a Docker image. This first iteration is large, non-optimized and uses Perl’s CPAN quite a bit too much for comfort, but I can show that it works. First, you need to pull in the image: $ docker pull kolab/jmap-proxySecond, you need to run it: $ docker run -d -p 80:80 kolab/jmap-proxyUnder the working assumption that you can now reach http://172.17.42.1/, you should therefore also have a JMAP proxy up and running. Enjoy!   [Less]