CubicWeb

You might have noticed here and there the mysterious user "mailbot" on cubicweb.org or logilab.org (both running the CubicWeb web app). Who is this user ? Well, one of the cool features about cubicweb is that you can interact with it simply by ... [More] using your email. When you are registered on a site, you can subscribe to a software project for example, from then on, you receive notifications of the new tickets and comments on the project. When you receive such a notification you can simply do an email reply to the new ticket or new comment, and cubicweb on the receiving end will import the content of your email to the website. When the content is imported that way, it's the mailbot doing the job. This is not rocket science, but it sure is useful. Follow the activity of the site by email and interact directly with comments and tickets from your mail client! image by husin.sani under creative commons [Less]

You might have noticed here and there the mysterious user "mailbot" on cubicweb.org or logilab.org (both running the CubicWeb web app). Who is this user ? .. image :: http://farm4.static.flickr.com/3244/2959912279_8446aa1abd_m.jpg :align: left ... [More] Well, one of the cool features about cubicweb is that you can interact with it simply by using your email. When you are registered on a site, you can subscribe to a software project for example, from then on, you receive notifications of the new tickets and comments on the project. When you receive such a notification you can simply do an email reply to the new ticket or new comment, and cubicweb on the receiving end will import the content of your email to the website. When the content is imported that way, it's the mailbot doing the job. This is not rocket science, but it sure is useful. Follow the activity of the site by email and interact directly with comments and tickets from your mail client! *image by* `husin.sani`_ *under* `creative commons`_ .. _`creative commons` : http://creativecommons.org/licenses/by/2.0/ .. _`husin.sani` : http://www.flickr.com/photos/9009139@N08/ [Less]

Today, I felt like doing a quick tour of the migration features provided by the ORMs used by the Python web frameworks. I started with Django. South looks better than Django-evolution which looks much better than dmigrations which is very low level. ... [More] I also had a look at SQLAlchemy.migrate, but again, that's too low level for me since I am looking to define migrations with the same vocabulary that is used for the data model, independently of the underlying database schema. The features listed in the South documentation have all been in CubicWeb for some time, except dependencies and autodetection. In my opinion, the dependency feature is not needed when you already have a list of scripts ordered by number, which is the case in South and in CubicWeb. The autodetection feature is more interesting, but it is tricky to get right. CubicWeb migration mechanism has had some kind of autodetection for a long time, but it is limited to the part that is easy to get right, yet quite common and useful: synchronizing properties of attributes and relationships (i.e. a Person.name becomes fulltextindexed or a has_portfolio relationship changes from 1-1 to 1-n) synchronizing permissions For other common tasks like adding or removing entities and attributes, high-level directives are provided like add_entity_type or remove_attribute. Up to now, not pushing autodetection of changes in the data model has been a deliberate choice, for diff'ing two models is complex and creating a migration path is even more difficult. Moreover, letting the ORM automatically overwrite local changes in the database schema can be harmful in some cases. In CubicWeb, the idea is that the developer knows better than the framework, so let him decide what's best and provide him with a concise vocabulary to write the migration scripts. photo by Tim in Sydney under creative commons. [Less]

Today, I felt like doing a quick tour of the migration features provided by the ORMs used by the Python web frameworks. I started with Django. South_ looks better than Django-evolution_ which looks much better than dmigrations_ which is very low ... [More] level. I also had a look at `SQLAlchemy.migrate`_, but again, that's too low level for me since I am looking to define migrations with the same vocabulary that is used for the data model, independently of the underlying database schema. .. image :: http://south.aeracode.org/raw-attachment/wiki/Logo/logo-trac.png :align: left The features listed in the South documentation have all been in CubicWeb for some time, except dependencies_ and autodetection_. In my opinion, the dependency feature is not needed when you already have a list of scripts ordered by number, which is the case in South and in CubicWeb. The autodetection feature is more interesting, but it is tricky to get right. CubicWeb migration mechanism has had some kind of autodetection for a long time, but it is limited to the part that is easy to get right, yet quite common and useful: * synchronizing properties of attributes and relationships (i.e. a Person.name becomes fulltextindexed or a has_portfolio relationship changes from 1-1 to 1-n) * synchronizing permissions .. image :: http://farm2.static.flickr.com/1007/666142945_1d675bc2a7_m.jpg :align: right For other common tasks like adding or removing entities and attributes, high-level directives are provided like `add_entity_type` or `remove_attribute`. Up to now, not pushing autodetection of changes in the data model has been a deliberate choice, for diff'ing two models is complex and creating a migration path is even more difficult. Moreover, letting the ORM automatically overwrite local changes in the database schema can be harmful in some cases. In CubicWeb, the idea is that the developer knows better than the framework, so let him decide what's best and provide him with a concise vocabulary to write the migration scripts. .. _South: http://south.aeracode.org/ .. _Django-evolution: http://code.google.com/p/django-evolution/ .. _dmigrations: http://code.google.com/p/dmigrations/ .. _dependencies: http://south.aeracode.org/wiki/Dependencies .. _autodetection: http://south.aeracode.org/wiki/Autodetection .. _`SQLAlchemy.migrate`: http://packages.python.org/sqlalchemy-migrate/ photo by `Tim in Sydney`_ under `creative commons`_. .. _`Tim in Sydney`: http://www.flickr.com/photos/tgillin/ .. _`creative commons`: http://creativecommons.org/licenses/by/2.0/ [Less]

.. image :: http://maps.google.com/intl/fr_ALL/images/maps_logo_small_blue.png :align: right There is this so-called '*gmap-view*' in CubicWeb, the question is: how to use it ? Well, first, no surprise, you have to generate an `API`_ key to ... [More] be able to use google maps on your server (make sure your usage conforms the terms as defined by Google). Now, let's say you have defined the following schema:: class Company(EntityType): name = String(required=True, maxsize=64) # ... some other attributes ... latitude = Float(required=True) longitude = Float(required=True) class Employee(EntityType): # ... some attributes ... works_for = SubjectRelation('Company', cardinality='1*') And you'd like to be able to display companies on a map; you've also got these nice icons that you'd wish to use as markers on the map. First thing, define those three icons as external resources. You can do that by editing your CUBE/data/external_resources file:: SMALL_MARKER_ICON=DATADIR/small_company.png MEDIUM_MARKER_ICON=DATADIR/MEDIUM_company.png BIG_MARKER_ICON=DATADIR/big_company.png We're nearly done, now. We just have to make our entity class implement the **cubicweb.interfaces.IGeocodable** interface. Here's an example:: from cubicweb.entities import AnyEntity from cubicweb.interfaces import IGeocodable class Company(AnyEntity): id = 'Company' # this must match the type as defined in your schema __implements__ = AnyEntity.__implements__ (IGeocodable,) def size(self): return self.req.execute('Any COUNT(E) WHERE E works_for C, C eid %(c)s', {'c': self.eid}) # this is a method of IGeocodable def marker_icon(self): size = self.size() if size < 20: return self.req_external_resource('SMALL_MARKER_ICON') elif size < 500: return self.req_external_resource('MEDIUM_MARKER_ICON') else: return self.req_external_resource('BIG_MARKER_ICON') That's it, you can now call the *gmap-view* on a resultset containing companies:: rset = self.req.execute('Any C WHERE C is Company') self.wview(rset, 'gmap-view', gmap_key=YOUR_API_KEY) Further configuration is possible, especially to control the size of the map or the default zoom level. To be fair, I must say that in a real-life cube, chances are you won't be able to specificy directly latitude and longitude and that you'll only have an address. This is slightly more complex to do since you'll need to query a geocoding service (the google one for instance) to transform your address into latitude/longitude. This will typically be done in a `hook`_ Here is an screenshot of google maps on a production site, the `museums in Normandy`_ : .. image :: http://www.cubicweb.org/image/229641?vid=download :align: center .. _API: http://code.google.com/intl/fr-FR/apis/maps/signup.html .. _hook: http://www.cubicweb.org/doc/en/B2020-hooks.en.html .. _`museums in Normandy` : http://collections.musees-haute-normandie.fr/collections/ [Less]

There is this so-called 'gmap-view' in CubicWeb, the question is: how to use it ? Well, first, no surprise, you have to generate an API key to be able to use google maps on your server (make sure your usage conforms the terms as defined by ... [More] Google). Now, let's say you have defined the following schema: class Company(EntityType): name = String(required=True, maxsize=64) # ... some other attributes ... latitude = Float(required=True) longitude = Float(required=True) class Employee(EntityType): # ... some attributes ... works_for = SubjectRelation('Company', cardinality='1*') And you'd like to be able to display companies on a map; you've also got these nice icons that you'd wish to use as markers on the map. First thing, define those three icons as external resources. You can do that by editing your CUBE/data/external_resources file: SMALL_MARKER_ICON=DATADIR/small_company.png MEDIUM_MARKER_ICON=DATADIR/MEDIUM_company.png BIG_MARKER_ICON=DATADIR/big_company.png We're nearly done, now. We just have to make our entity class implement the cubicweb.interfaces.IGeocodable interface. Here's an example: from cubicweb.entities import AnyEntity from cubicweb.interfaces import IGeocodable class Company(AnyEntity): id = 'Company' # this must match the type as defined in your schema __implements__ = AnyEntity.__implements__ (IGeocodable,) def size(self): return self.req.execute('Any COUNT(E) WHERE E works_for C, C eid %(c)s', {'c': self.eid}) # this is a method of IGeocodable def marker_icon(self): size = self.size() if size < 20: return self.req_external_resource('SMALL_MARKER_ICON') elif size < 500: return self.req_external_resource('MEDIUM_MARKER_ICON') else: return self.req_external_resource('BIG_MARKER_ICON') That's it, you can now call the gmap-view on a resultset containing companies: rset = self.req.execute('Any C WHERE C is Company') self.wview(rset, 'gmap-view', gmap_key=YOUR_API_KEY) Further configuration is possible, especially to control the size of the map or the default zoom level. To be fair, I must say that in a real-life cube, chances are you won't be able to specificy directly latitude and longitude and that you'll only have an address. This is slightly more complex to do since you'll need to query a geocoding service (the google one for instance) to transform your address into latitude/longitude. This will typically be done in a hook Here is an screenshot of google maps on a production site, the museums in Normandy : [Less]

.. image :: http://www.cubicweb.org/image/212907?vid=download :align: left Here is a brief summary of what you get for the new CubicWeb 3.1.0_ release. You could obviously go though the tickets on the version_ page, but here is the short ... [More] version. What new features ? * a few OWL_ and Linked_Data_ functionalities * navigation is now more complete on search results * when installing a new cube that requires anonymous access (public site) the installer enables that access What bugs are fixed ? * a few things didn't work with opera and IE6 * json controller conflicts solved * the newcube command is working again * facets don't get in the way of the association process anymore * and more... Hope you enjoy this version... to see what's coming next, you can check out the planned versions of CubicWeb : 3.1.1_ and 3.2.0_. .. _version : http://www.cubicweb.org/project/cubicweb/3.1.0 .. _3.1.0 : http://www.cubicweb.org/project/cubicweb/3.1.0 .. _OWL : http://en.wikipedia.org/wiki/Web_Ontology_Language .. _Linked_Data : http://en.wikipedia.org/wiki/Linked_Data .. _3.1.1 : http://www.cubicweb.org/project/cubicweb/3.1.1 .. _3.2.0 : http://www.cubicweb.org/project/cubicweb/3.2.0 [Less]

Recently, for internal purposes, we've made a little cubicweb application to help us organizing visits to find new office locations. Here's an *excerpt* of the schema:: class Office(WorkflowableEntityType): price = ... [More] Int(description='euros / m2 / HC / HT') surface = Int(description='m2') description = RichString(fulltextindexed=True) has_address = SubjectRelation('PostalAddress', cardinality='1?', composite='subject') proposed_by = SubjectRelation('Agency') comments = ObjectRelation('Comment', cardinality='1*', composite='object') screenshots = SubjectRelation(('File', 'Image'), cardinality='*1', composite='subject') The two other entity types defined in the schema are `Visit` and `Agency` but we can also guess from the above that this application uses the two cubes `comment`_ and `addressbook`_ (remember, cubicweb is only a game where you assemble cubes !). While we know that just defining the schema in enough to have a full, usable, (testable !) application, we also know that every application needs to be customized to fulfill the needs it was built for. So in this case, what we needed most was some custom filters that would let us restrict searches according to surfaces, prices or zipcodes. Fortunately for us, Cubicweb provides the **facets** (image_) mechanism and a few base classes that make the task quite easy:: class PostalCodeFacet(RelationFacet): id = 'postalcode-facet' # every registered class must have an id __select__ = implements('Office') # this facet should only be selected when # visualizing offices rtype = 'has_address' # this facet is a filter on the entity linked to # the office thrhough the relation has_address target_attr = 'postalcode' # the filter's key is the attribute "postal_code" # of the target PostalAddress entity This is a typical `RelationFacet`: we want to be able to filter offices according to the attribute `postalcode` of their associated `PostalAdress`. Each line in the class is explained by the comment on its right. Now, here is the code to define a filter based on the `surface` attribute of the `Office`:: class SurfaceFacet(AttributeFacet): id = 'surface-facet' # every registered class must have an id __select__ = implements('Office') # this facet should only be selected when # visualizing offices rtype = 'surface' # the filter's key is the attribute "surface" comparator = '>=' # override the default value of operator since # we want to filter according to a minimal # value, not an exact one def rset_vocabulary(self, ___): """override the default vocabulary method since we want to hard-code our threshold values. Not overriding would generate a filter box with all existing surfaces defined in the database. """ return [('> 200', '200'), ('> 250', '250'), ('> 275', '275'), ('> 300', '300')] And that's it: we have two filter boxes automatically displayed on each page presenting more than one office. The `price` facet is basically the same as the `surface` one but with a different vocabulary and with ``rtype = 'price'``. (The cube also benefits from the builtin google map views defined by cubicweb but that's for another blog). .. _image: http://www.cubicweb.org/image/197646?vid=download .. _comment: http://www.cubicweb.org/project/cubicweb-comment .. _addressbook: http://www.cubicweb.org/project/cubicweb-addressbook [Less]

Recently, for internal purposes, we've made a little cubicweb application to help us organizing visits to find new office locations. Here's an excerpt of the schema: class Office(WorkflowableEntityType): price = Int(description='euros / m2 / ... [More] HC / HT') surface = Int(description='m2') description = RichString(fulltextindexed=True) has_address = SubjectRelation('PostalAddress', cardinality='1?', composite='subject') proposed_by = SubjectRelation('Agency') comments = ObjectRelation('Comment', cardinality='1*', composite='object') screenshots = SubjectRelation(('File', 'Image'), cardinality='*1', composite='subject') The two other entity types defined in the schema are Visit and Agency but we can also guess from the above that this application uses the two cubes comment and addressbook (remember, cubicweb is only a game where you assemble cubes !). While we know that just defining the schema in enough to have a full, usable, (testable !) application, we also know that every application needs to be customized to fulfill the needs it was built for. So in this case, what we needed most was some custom filters that would let us restrict searches according to surfaces, prices or zipcodes. Fortunately for us, Cubicweb provides the facets (image) mechanism and a few base classes that make the task quite easy: class PostalCodeFacet(RelationFacet): id = 'postalcode-facet' # every registered class must have an id __select__ = implements('Office') # this facet should only be selected when # visualizing offices rtype = 'has_address' # this facet is a filter on the entity linked to # the office thrhough the relation has_address target_attr = 'postalcode' # the filter's key is the attribute "postal_code" # of the target PostalAddress entity This is a typical RelationFacet: we want to be able to filter offices according to the attribute postalcode of their associated PostalAdress. Each line in the class is explained by the comment on its right. Now, here is the code to define a filter based on the surface attribute of the Office: class SurfaceFacet(AttributeFacet): id = 'surface-facet' # every registered class must have an id __select__ = implements('Office') # this facet should only be selected when # visualizing offices rtype = 'surface' # the filter's key is the attribute "surface" comparator = '>=' # override the default value of operator since # we want to filter according to a minimal # value, not an exact one def rset_vocabulary(self, ___): """override the default vocabulary method since we want to hard-code our threshold values. Not overriding would generate a filter box with all existing surfaces defined in the database. """ return [('> 200', '200'), ('> 250', '250'), ('> 275', '275'), ('> 300', '300')] And that's it: we have two filter boxes automatically displayed on each page presenting more than one office. The price facet is basically the same as the surface one but with a different vocabulary and with rtype = 'price'. (The cube also benefits from the builtin google map views defined by cubicweb but that's for another blog). [Less]

In test driven developpement (TDD_), you write the test before you write the code. On a web application, number of levels can be tested. Here are a few hints at how we manage some of the testing with CubicWeb. We use pytest_ (which is an ... [More] extension of python's unittest framework available in logilab-common_) to execute all tests across the cubes. Even in the core of cubicweb the tests are spread out across the server, web part, repository, common tools... so a simple pytest_ command crawls though all theses tests and runs them. .. image:: http://www.sqlite.org/images/SQLite.gif :align: right **The problem** : One of the tricky things with testing CubicWeb is that the structure of the data is imported into the database (which enables us to easily modify the schema on running data), and that test data can be long to generate and fake for a web application that is used to talk to a proper database server (postgres_). So we though of inserting test data into an sqlite_ database. After a bit of work on compatibility, it was up an running. But setting up that database was (and still is) quite long, testing was becoming way too long, TDD (with frequent testing) was becoming impossible. **The solution** : we ended up storing the sqlite_ database in a temporary file which is used up if it's not too old, TDD was back in the loop. So if you're developing for CubicWeb don't worry about those *test/tmpdb* files, on the contrary, that means you're running tests. For writing tests, check out the content about it in the book_. .. _pytest : http://www.logilab.org/project/logilab-common .. _TDD : http://en.wikipedia.org/wiki/Test-driven_development .. _logilab-common : http://www.logilab.org/project/logilab-common .. _postgres : http://www.postgresql.org .. _sqlite : http://www.sqlite.org .. _book: http://www.cubicweb.org/doc/en/B3010-tests.en.html [Less]