Posted
about 4 years
ago
by
Nicolas Chauvat
Hello CubicWeb community,
We are pleased to announce the release of CubicWeb 3.27. Many thanks to
all the contributors of this release!
Main changes in this release are listed below. Please note this release drops python2 support.
Enjoy this new
... [More]
version!
New features
Tests can now be run concurrently across multiple processes. You can use
pytest-xdist for that. For tests using PostgresApptestConfiguration you
should be aware that startpgcluster() can't run concurrently. Workaround is
to call pytest with --dist=loadfile to use a single test process per test
module or use an existing database cluster and set db-host and
db-port of devtools.DEFAULT_PSQL_SOURCES['system'] accordingly.
on cubicweb-ctl create and cubicweb-ctl pyramid, if it doesn't already
exist in the instance directory, the pyramid.ini file will be generated
with the needed secrets.
add a --pdb flag to all cubicweb-ctl command to launch (i)pdb if an exception
occurs during a command execution.
the --loglevel and --dbglevel flags are available for all cubicweb-ctl
instance commands (and not only the pyramid one)
following "only in foreground" behavior all commands logs to stdout by
default from now on. To still log to a file pass log_to_file=True to
CubicWebConfiguration.config_for
add a new migration function update_bfss_path(old_path, new_path) to update
the path in Bytes File-System Storage (bfss).
on every request display request path and selected controller in CLI
migration interactive mode improvements:
when an exception occurs, display the full traceback instead of only the exception
on migration p(db) choice, launch ipdb if it's installed
on migration p(db) choice, give the traceback to pdb if it's available,
this mean that the (i)pdb interactive session will be on the stack of
the exception instead of being on the stack where pdb is launched which
will allow the user to access all the relevant context of the exception
which otherwise is lost
on DBG_SQL and/or DBG_RQL, if pygments is installed, syntax highlight sql/rql
debug output
allow to specify the instance id for any instance command using the
CW_INSTANCE global variable instead of or giving it as a cli argument
when debugmode is activated ('-D/--debug' on the pyramid command for
example), the HTML generated by CW will contains new tags that will indicate
by which object in the code it has been generated and in which line of which
source code.
For example:
unset title
[...]
While this hasn't been done yet, this feature is an open path for building dynamic tools that can help inspect the page.
a new debug channels mechanism has been added, you can subscribe to one of
those channels in your python code to build debug tools for example (the
pyramid custom panels are built using that) and you will receive a
datastructure (a dict) containing related information. The available channels
are: controller, rql, sql, vreg, registry_decisions
add a new '-t/--toolbar' option the pyramid command to activate the pyramid debugtoolbar
a series of pyramid debugtoolbar panels specifically made for CW, see bellow
Pyramid debugtoolbar and custom panel
The pyramid debugtoolbar is now integrated into CubicWeb during the development
phase when you use the 'pyramid' command. To activate it you need to pass the
'-t/--toolbar' argument to the 'pyramid' command.
In addition, a series of custom panels specifically done for CW are now
available, they display useful information for the development and the
debugging of each page. The available panels are:
a general panel which contains the selected controller, the current
settings and useful links screenshot1
a panel listing all decisions taken in registry for building this page screenshot2
a panel listing the content of the vreg registries screenshot3
a panel listing all the RQL queries made during a request screenshot4
a panel listing all the SQL queries made during a request screenshot5
Furthermore, in all those panels, next to each object/class/function/method a
link to display its source code is available (shown as '[source]' screenshot6) and also every file path shown
is a traceback is also a link to display the corresponding file (screenshot7). For example: screenshot8.
Backwards incompatible changes
Standardization on the way to launch a cubicweb instance, from now on the
only way to do that will be the used the pyramid command. Therefore:
cubicweb-ctl commands "start", "stop", "restart", "reload" and "status"
have been removed because they relied on the Twisted web server backend that
is no longer maintained nor working with Python 3.
Twisted web server support has been removed.
cubicweb-ctl wsgi has also been removed.
Support for legacy cubes (in the 'cubes' python namespace) has been dropped.
Use of environment variables CW_CUBES_PATH and CUBES_DIR is removed.
Python 2 support has been dropped.
Exceptions in notification hooks aren't catched-all anymore during tests so
one can expect tests that seem to pass (but were actually silently failing)
to fail now.
All "cubicweb-ctl" command only accept one instance argument from now one
(instead of 0 to n)
'pyramid' command will always run in the foreground now, by consequence the
option --no-daemon has been removed.
DBG_MS flag has been removed since it is not used anymore
transactions db logs where displayed using the logging
(debug/info/warning...) mechanism, now it is only displayed if the
corresponding DBG_OPS flag is used
Deprecated code drops
Most code deprecated by version 3.25 or older versions has been dropped.
[Less]
|
Posted
over 4 years
ago
by
Laurent Peuch
One of our next project for cubicweb and its ecosystem is to implement
the langserver protocol for the RQL language that we are using to query the data stored in CubicWeb.
The langserver protocol is an idea to solve one problem: to integrate
... [More]
operation for various languages, most IDE/tools needs to reimplement
the wheel all the time, doing custom plugin etc... To solve this
issue, this protocol has been invented with one idea: make one server
for a language, then all IDE/tools that talks this protocol will be
able to integrate it easily.
So the idea is simple: let's build our own server for RQL so we'll be
able to integrate it everywhere and build tools for it.
Since RQL has similarities with GraphQL, one of the goals is to have something similar to Graphiql which is for example used by GitHub to expose their API at
https://developer.github.com/v4/explorer/
So this post has several objectives:
gather people that would be motivate to work on that subject, for now
there is Laurent Wouters and me :)
explain to you in more details (not all) how the language server
protocol works
show what is already existing for both langserver in python and rql
show the first roadmap we've discussed with Laurent Wouters on how
we think we can do that :)
be a place to discuss this project, things aren't fixed yet :)
So, what is the language server protocol (LSP)?
It's a JSON-RPC based protocol where the IDE/tool talks to the server.
JSON-RPC, said simply, is a bi-directional protocol in json.
In this procotol you have 2 kind of exchanges:
requests: where the client (or server) ask the server (or the server
ask the client) something and a reply is expected. For example:
where is the definition of this function?
notifications: the same but without an expected reply. For example:
linting information or error detection
The LSP specifications has 3 bigs categories:
everything about initialization/shutdown the server etc...
everything regarding text and workspace synchronization between the
server and the client
the actual things that interest us: a list of languages features
that the server supports (you aren't in the obligation to implement
everything)
Here is the simplified list of possible languages features that the
website present:
Code completion
Hover
Jump to def
Workspace symbols
Find references
Diagnostics
The specification is much more detailed but way less comprehensive
(look at the "language features" on the right menu for more details):
completion/completion resolve
hover (when you put your cursor on something)
signatureHelp
declaration (go to...)
definition (go to...)
typeDefinition (go to...)
implementation (go to...)
references
documentHighlight (highlight all references to a symbol)
documentSymbol ("symbol" is a generic term for variable, definitions etc...)
codeAction (this one is interesting)
codeLens/codeLens resolve
documentLink/documentLink resolve
documentColor/colorPresentation (stuff about picking colors)
formatting/rangeFormatting/onTypeFormatting (set tab vs space)
rename/prepareRename
foldingRange
(Comments are from my current understanding of the spec, it might not
be perfect)
The one that is really interesting here (but not our priority right
now) is "codeAction", it's basically a generic entry point for every
refactoring kind of operations as some examples from the spec shows:
Example extract actions:
Extract method
Extract function
Extract variable
Extract interface from class
Example inline actions:
Inline function
Inline variable
Inline constant
Example rewrite actions:
Convert JavaScript function to class
Add or remove parameter
Encapsulate field
Make method static
Move method to base class
But I'm not expecting us to have direct need for it but that really
seems one to keep in mind.
One question that I frequently got was: is syntax highlight included
in the langserver protocol? Having double checked with Laurent
Wouters, it's actually not the case (I thought documentSymbol could be
used for that but actually no).
But we already have an implementation for that in pygments:
https://hg.logilab.org/master/rql/file/d30c34a04ebf/rql/pygments_ext.py
What is currently existing for LSP in python and rql
The state is not great in the python ecosystem but not a disaster.
Right now I haven't been able to find any generic python
implementation of LSP that we could really reuse and integrate.
There is, right now and to my knowledge, only 2 maintained
implementation of LSP in python. One for python and one for ...
Fortran x)
https://github.com/palantir/python-language-server
https://github.com/hansec/fortran-language-server
Palantir's one makes extensive use of advanced magic code doesn't seems
really necessary but it is probably of higher quality code since the Fortran
one doesn't seems very idiomatic but looks much simpler.
So we'll ever need to extract the needed code from one of those of
implement our own, not so great.
On the RQL side, everything that seems to be useful for our current
situation is located in the RQL package that we maintain:
https://hg.logilab.org/master/rql
Roadmap
After a discussion with Laurent Wouters, a first roadmap looks like
this:
extract the code from either palantir or fortran LSP implementation
and come with a generic implementation (I'm probably going to do it
but Laurent told me he his going to take a look too)
When I'm talking about a generic implementation I'm talking about
everything listed in the big category of the protocol that isn't
related to language features which we don't really want to rewrite
again.
Once that's done, start implementing the language features for RQL:
the easiest is the syntax errors detection code, we just need to
launch to parser on the code and handle the potential errors
do that with pretty specific red underline
play with RQL AST to extract the symbols and start doing things like
codeLens and hover
much more complex (and for later): autocompletion (we'll either need
a demi compiler or to modify the current one for that)
Side note
To better understand the motivation behind this move, it is part of the more
global move of drop the "Web" from CubicWeb and replace all the front end
current implementation by reactjs+typescript views. In this context CubicWeb
(or Cubic?) will only serves as a backend provide with which we will talk in... RQL!
Therefor writing and using RQL will be much more important than right now. [Less]
|
Posted
over 4 years
ago
by
Laurent Peuch
One of our next project for cubicweb and its ecosystem is to implement
the langserver protocol for the RQL language that we are using in CW.
The langserver protocol is an idea to solve one problem: to integrate
operation for various languages, most
... [More]
IDE/tools needs to reimplement
the wheel all the time, doing custom plugin etc... To solve this
issue, this protocol has been invented with one idea: make one server
for a language, then all IDE/tools that talks this protocol will be
able to integrate it easily.
You can find the website here: https://langserver.org/
So the idea is simple: let's build our own server for RQL so we'll be
able to integrate it everywhere and build tools for it.
One of the goal is to have something similar than that for RQL:
https://developer.github.com/v4/explorer/ rql being extremely similar to graphql
So this post has several objectives:
gather people that would be motivate to work on that subject, for now
there is Laurent Wouters and me :)
explain to you in more details (not all) how the language server
protocol works
show what is already existing for both langserver in python and rql
show the first roadmap we've discussed with Laurent Wouters on how
we think we can do that :)
be a place to discuss this project, things aren't fixed yet :)
So, what is the language server protocol (LSP)?
It's a JSON-RPC based protocol where the IDE/tool talks to the server.
JSON-RPC, said simply, is a bi-directional protocol in json.
In this procotol you have 2 kind of exchanges:
requests: where the client (or server) ask the server (or the server
ask the client) something and a reply is expected. For example:
where is the definition of this function?
notifications: the same but without an expected reply. For example:
linting information or error detection
The LSP specifications has 3 bigs categories:
everything about initialization/shutdown the server etc...
everything regarding text and workspace synchronization between the
server and the client
the actual things that interest us: a list of languages features
that the server supports (you aren't in the obligation to implement
everything)
Here is the simplified list of possible languages features that the
website present:
Code completion
Hover
Jump to def
Workspace symbols
Find references
Diagnostics
The specification is much more detailed but way less comprehensive
(look at the "language features" on the right menu for more details):
completion/completion resolve
hover (when you put your cursor on something)
signatureHelp
declaration (go to...)
definition (go to...)
typeDefinition (go to...)
implementation (go to...)
references
documentHighlight (highlight all references to a symbol)
documentSymbol ("symbol" is a generic term for variable, definitions etc...)
codeAction (this one is interesting)
codeLens/codeLens resolve
documentLink/documentLink resolve
documentColor/colorPresentation (stuff about picking colors)
formatting/rangeFormatting/onTypeFormatting (set tab vs space)
rename/prepareRename
foldingRange
(Comments are from my current understanding of the spec, it might not
be perfect)
The one that is really interesting here (but not our priority right
now) is "codeAction", it's basically a generic entry point for every
refactoring kind of operations as some examples from the spec shows:
Example extract actions:
- Extract method
- Extract function
- Extract variable
- Extract interface from class
Example inline actions:
Inline function
Inline variable
Inline constant
Example rewrite actions:
Convert JavaScript function to class
Add or remove parameter
Encapsulate field
Make method static
Move method to base class
But I'm not expecting us to have direct need for it but that really
seems one to keep in mind.
One question that I frequently got was: is syntax highlight included
in the langserver protocol? Having double checked with Laurent
Wouters, it's actually not the case (I thought documentSymbol could be
used for that but actually no).
But we already have an implementation for that in pygments:
https://hg.logilab.org/master/rql/file/d30c34a04ebf/rql/pygments_ext.py
What is currently existing for LSP in python and rql
The state is not great in the python ecosystem but not a disaster.
Right now I haven't been able to find any generic python
implementation of LSP that we could really reuse and integrate.
There is, right now and to my knowledge, only 2 maintained
implementation of LSP in python. One for python and one for ...
Fortran x)
https://github.com/palantir/python-language-server
https://github.com/hansec/fortran-language-server
Palantir's one makes extensive use of advanced magic code doesn't seems
really necessary but it is probably of higher quality code since the Fortran
one doesn't seems very idiomatic but looks much simpler.
So we'll ever need to extract the needed code from one of those of
implement our own, not so great.
On the RQL side, everything that seems to be useful for our current
situation is located in the RQL package that we maintain:
https://hg.logilab.org/master/rql
Roadmap
After a discussion with Laurent Wouters, a first roadmap looks like
this:
extract the code from either palantir or fortran LSP implementation
and come with a generic implementation (I'm probably going to do it
but Laurent told me he his going to take a look too)
When I'm talking about a generic implementation I'm talking about
everything listed in the big category of the protocol that isn't
related to language features which we don't really want to rewrite
again.
Once that's done, start implementing the language features for RQL:
the easiest is the syntax errors detection code, we just need to
launch to parser on the code and handle the potential errors
do that with pretty specific red underline
play with RQL AST to extract the symbols and start doing things like
codeLens and hover
much more complex (and for later): autocompletion (we'll either need
a demi compiler or to modify the current one for that)
Side note
To better understand the motivation behind this move, it is part of the more
global move of drop the "Web" from CubicWeb and replace all the front end
current implementation by reactjs+typescript views. In this context CubicWeb
(or Cubic?) will only serves as a backend provide with which we will talk in... RQL!
Therefor writing and using RQL will be much more important than right now. [Less]
|
Posted
about 7 years
ago
by
Denis Laxalde
This is the second post of a series about cubicweb-jsonschema. The first
post mainly dealt with JSON Schema representations of CubicWeb entities
along with a brief description of the JSON API. In this second post, I'll
describe another aspect of the
... [More]
project that aims at building an hypermedia API
by leveraging the JSON Hyper Schema specification.
Hypermedia APIs and JSON Hyper Schema
Hypermedia API is somehow a synonymous of RESTful API but it makes it clearer
that the API serves hypermedia responses, i.e. content that helps
discoverability of other resources.
At the heart of an hypermedia API is the concept of link relation which both aims at describing
relationships between resources as well as provinding ways to manipulate them.
In JSON Hyper Schema terminology, link relations take the form of a collection
of Link Description Objects gathered into a links property of a JSON
Schema document. These Link Description Objects thus describes relationships
between the instance described by the JSON Schema document at stake and
other resources; they hold a number of properties that makes relationships
manipulation possible:
rel is the name of the relation, it is usually one of relation names
registered at IANA;
href indicates the URI of the target of the relation, it may be
templated by a JSON Schema;
targetSchema is a JSON Schema document (or reference) describing the
target of the link relation;
schema (recently renamed as submissionSchema) is a JSON Schema
document (or reference) describing what the target of the link expects when
submitting data.
Hypermedia walkthrough
In the remaining of the article, I'll walk through a navigation path that is
made possible by hypermedia controls provided by cubicweb-jsonschema. I'll
continue on the example application described in the first post of the
series which schema consists of Book, Author and Topic entity types. In
essence, this walkthrough is typical of what an intelligent client could do
when exposed to the API, i.e. from any resource, discover other resources and
navigate or manipulate them.
This walkthrough assumes that, given any resource (i.e. something that has a
URL like /book/1), the server would expose data at the main URL when the
client asks for JSON through the Accept header and it would expose the
JSON Schema of the resource at a schema view of the same URL (i.e.
/book/1/schema). This assumption can be regarded as a kind of
client/server coupling, which might go away in later implementation.
Site root
While client navigation could start from any resource, we start from the
root resource and retrieve its schema:
GET /schema
Accept: application/schema+json
HTTP/1.1 200 OK
Content-Type: application/json
{
"links": [
{
"href": "/author/",
"rel": "collection",
"schema": {
"$ref": "/author/schema?role=creation"
},
"targetSchema": {
"$ref": "/author/schema"
},
"title": "Authors"
},
{
"href": "/book/",
"rel": "collection",
"schema": {
"$ref": "/book/schema?role=creation"
},
"targetSchema": {
"$ref": "/book/schema"
},
"title": "Books"
},
{
"href": "/topic/",
"rel": "collection",
"schema": {
"$ref": "/topic/schema?role=creation"
},
"targetSchema": {
"$ref": "/topic/schema"
},
"title": "Topics"
}
]
}
So at root URL, our application serves a JSON Hyper Schema that only consists
of links. It has no JSON Schema document, which is natural since there's
usually no data bound to the root resource (think of it as empty rset in
CubicWeb terminology).
These links correspond to top-level entity types, i.e. those that would
appear in the default startup page of a CubicWeb application. They all have
"rel": "collection" relation name (this comes from RFC6573) as their
target is a collection of entities. We also have schema and
targetSchema properties.
Links as actions
So let's pick one of these links, the Books link. We can either just
follow the link URI href="/book/" using a GET request, but what if we
wanted to create a new book? We'd need to perform a POST request at
href URI with a body that conforms to the JSON Schema of schema
property. So let's first retrieve this schema:
GET /book/schema?role=creation
Accept: application/schema+json
HTTP/1.1 200 OK
Content-Length: 836
Content-Type: application/json
Date: Tue, 04 Apr 2017 09:46:53 GMT
Server: waitress
{
"$ref": "#/definitions/Book",
"definitions": {
"Book": {
"additionalProperties": false,
"properties": {
"author": {
"items": {
"oneOf": [
{
"enum": [
"856"
],
"title": "Ernest Hemingway"
},
{
"enum": [
"855"
],
"title": "Victor Hugo"
}
],
"type": "string"
},
"maxItems": 1,
"minItems": 1,
"title": "author",
"type": "array"
},
"publication_date": {
"format": "date-time",
"title": "publication date",
"type": "string"
},
"title": {
"title": "title",
"type": "string"
}
},
"required": [
"title",
"publication_date"
],
"title": "Book",
"type": "object"
}
}
}
(This is essentially the same schema that we had in the last example of the
first post.)
With this we may form a POST request with a JSON document as body that
conforms to this schema, but how do we know that we can actually send the
request? Hypermedia link relations do not provide any information for this.
Instead, we rely on the underlying protocol (HTTP) and on the server to
advertize its capabilities. This means we need to fetch the target ressource
and inspect the Allow header in the response. Most of the times, we would
already have fetched the resource (verb GET) so these headers would be
readily available. But in this case, we are on the root resource and we have
not fetched targets of the links, so we use a HEAD request:
HEAD /book/
HTTP/1.1 200 OK
Allow: GET, POST
Verbs listed in this Allow header just describe what actions we are
allowed to perform on the resource. Under the hood, this is determined by
permissions lookup (i.e. "read" permission matches with GET verb, "add"
permission matches with POST verb and so on).
So we can create a Book!
POST /book/
Accept: application/json
{"title": "L'homme qui rit", "author": ["855"], "publication_date": "1869-04-01"}
HTTP/1.1 201 Created
Content-Type: application/json; charset=UTF-8
Location: http://localhost:6543/Book/859
{
"author": [
"Victor Hugo"
],
"publication_date": "1869-04-01",
"title": "L'homme qui rit"
}
What's important in this response is the Location header which indicates
where our new resource lives. Our intelligent client can then proceed with its
navigation using this information.
From collection to items
Now that we have added a new book, let's step back and use our books link to
retrieve data (verb GET):
GET /book/
Accept: application/json
HTTP/1.1 200 OK
Allow: GET, POST
Content-Type: application/json
[
{
"id": "859",
"title": "L'homme qui rit"
},
{
"id": "858",
"title": "The Old Man and the Sea"
},
]
which, as always, needs to be completed by a JSON Schema:
GET /book/schema
Accept: application/schema+json
HTTP/1.1 200 OK
Content-Type: application/json
{
"$ref": "#/definitions/Book_plural",
"definitions": {
"Book_plural": {
"items": {
"properties": {
"id": {
"type": "string"
},
"title": {
"type": "string"
}
},
"type": "object"
},
"title": "Books",
"type": "array"
}
},
"links": [
{
"href": "/book/",
"rel": "collection",
"schema": {
"$ref": "/book/schema?role=creation"
},
"targetSchema": {
"$ref": "/book/schema"
},
"title": "Books"
},
{
"href": "/book/{id}",
"rel": "item",
"targetSchema": {
"$ref": "/book/schema?role=view"
},
"title": "Book"
}
]
}
Consider the last item of links in the above schema. It has a "rel":
"item" property which indicates how to access items of the collection;
its href property is a templated URI which can be expanded using
instance data and schema (here we only have a single id template
variable).
So our client may navigate to the first item of the collection (id="859")
at /book/859 URI, and retrieve resource data:
GET /book/859
Accept: application/json
HTTP/1.1 200 OK
Allow: GET, PUT, DELETE
Content-Type: application/json
{
"author": [
"Victor Hugo"
],
"publication_date": "1869-04-01T00:00:00",
"title": "L'homme qui rit"
}
and schema:
GET /book/859/schema
Accept: application/schema+json
HTTP/1.1 200 OK
Content-Type: application/json
{
"$ref": "#/definitions/Book",
"definitions": {
"Book": {
"additionalProperties": false,
"properties": {
"author": {
"items": {
"type": "string"
},
"title": "author",
"type": "array"
},
"publication_date": {
"format": "date-time",
"title": "publication date",
"type": "string"
},
"title": {
"title": "title",
"type": "string"
},
"topics": {
"items": {
"type": "string"
},
"title": "topics",
"type": "array"
}
},
"title": "Book",
"type": "object"
}
},
"links": [
{
"href": "/book/",
"rel": "up",
"targetSchema": {
"$ref": "/book/schema"
},
"title": "Book_plural"
},
{
"href": "/book/859/",
"rel": "self",
"schema": {
"$ref": "/book/859/schema?role=edition"
},
"targetSchema": {
"$ref": "/book/859/schema?role=view"
},
"title": "Book #859"
}
]
}
Entity resource
The resource obtained above as an item of a collection is actually an entity.
Notice the rel="self" link. It indicates how to manipulate the current
resource (i.e. at which URI, using a given schema depending on what actions we
want to perform). Still this link does not indicate what actions may be
performed. This indication is found in the Allow header of the data
response above:
Allow: GET, PUT, DELETE
With these information bits, our intelligent client is able to, for instance,
form a request to delete the resource. On the other hand, the action to
update the resource (which is allowed because of the presence of PUT in
Allow header, per HTTP semantics) would take the form of a request which
body conforms to the JSON Schema pointed at by the schema property of the
link.
Also note the rel="up" link which makes it possible to navigate to the
collection of books.
Conclusions
This post introduced the main hypermedia capabilities of
cubicweb-jsonschema, built on top of the JSON Hyper Schema
specification. The resulting Hypermedia API makes it possible for an
intelligent client to navigate through hypermedia resources and manipulate
them by using both link relation semantics and HTTP verbs.
In the next post, I'll deal with relationships description and manipulation
both in terms of API (endpoints) and hypermedia representation.
[Less]
|
Posted
about 7 years
ago
by
Denis Laxalde
This is the first post of a series introducing the cubicweb-jsonschema
project that is currently under development at Logilab. In this post, I'll
first introduce the general goals of the project and then present in more
details two aspects about data
... [More]
models (the connection between Yams and JSON
schema in particular) and the basic features of the API. This post does not
always present how things work in the current implementation but rather how
they should.
Goals of cubicweb-jsonschema
From a high level point of view, cubicweb-jsonschema addresses mainly two
interconnected aspects. One related to modelling for client-side development
of user interfaces to CubicWeb applications while the other one concerns the
HTTP API.
As far as modelling is concerned, cubicweb-jsonschema essentially aims at
providing a transformation mechanism between a Yams schema and JSON Schema
that is both automatic and extensible. This means that we can ultimately
expect that Yams definitions alone would sufficient to have generated JSON
schema definitions that would consistent enough to build an UI, pretty much as
it is currently with the automatic web UI in CubicWeb. A corollary of this
goal is that we want JSON schema definitions to match their context of usage,
meaning that a JSON schema definition would not be the same in the context of
viewing, editing or relationships manipulations.
In terms of API, cubicweb-jsonschema essentially aims at providing an HTTP
API to manipulate entities based on their JSON Schema definitions.
Finally, the ultimate goal is to expose an hypermedia API for a CubicWeb
application in order to be able to ultimately build an intelligent client. For
this we'll build upon the JSON Hyper-Schema specification. This aspect will
be discussed in a later post.
Basic usage as an HTTP API library
Consider a simple case where one wants to manipulate entities of type Author
described by the following Yams schema definition:
class Author(EntityType):
name = String(required=True)
With cubicweb-jsonschema one can get JSON Schema for this entity type in at
different contexts such: view, creation or edition. For instance:
in a view context, the JSON Schema will be:
{
"$ref": "#/definitions/Author",
"definitions": {
"Author": {
"additionalProperties": false,
"properties": {
"name": {
"title": "name",
"type": "string"
}
},
"title": "Author",
"type": "object"
}
}
}
whereas in creation context, it'll be:
{
"$ref": "#/definitions/Author",
"definitions": {
"Author": {
"additionalProperties": false,
"properties": {
"name": {
"title": "name",
"type": "string"
}
},
"required": [
"name"
],
"title": "Author",
"type": "object"
}
}
}
(notice, the required keyword listing name property).
Such JSON Schema definitions are automatically generated from Yams
definitions. In addition, cubicweb-jsonschema exposes some endpoints for basic
CRUD operations on resources through an HTTP (JSON) API. From the client point
of view, requests on these endpoints are of course expected to match JSON
Schema definitions.
Some examples:
Get an author resource:
GET /author/855
Accept:application/json
HTTP/1.1 200 OK
Content-Type: application/json
{"name": "Ernest Hemingway"}
Update an author:
PATCH /author/855
Accept:application/json
Content-Type: application/json
{"name": "Ernest Miller Hemingway"}
HTTP/1.1 200 OK
Location: /author/855/
Content-Type: application/json
{"name": "Ernest Miller Hemingway"}
Create an author:
POST /author
Accept:application/json
Content-Type: application/json
{"name": "Victor Hugo"}
HTTP/1.1 201 Created
Content-Type: application/json
Location: /Author/858
{"name": "Victor Hugo"}
Delete an author:
DELETE /author/858
HTTP/1.1 204 No Content
Now if the client sends invalid input with respect to the schema, they'll get
an error:
(We provide a wrong born property in request body.)
PATCH /author/855
Accept:application/json
Content-Type: application/json
{"born": "1899-07-21"}
HTTP/1.1 400 Bad Request
Content-Type: application/json
{
"errors": [
{
"details": "Additional properties are not allowed ('born' was unexpected)",
"status": 422
}
]
}
From Yams model to JSON Schema definitions
The example above illustrates automatic generation of JSON Schema documents
based on Yams schema definitions. These documents are expected to help
developping views and forms for a web client. Clearly, we expect that
cubicweb-jsonschema serves JSON Schema documents for viewing and editing
entities as cubicweb.web serves HTML documents for the same purposes. The
underlying logic for JSON Schema generation is currently heavily inspired by
the logic of primary view and automatic entity form as they exists in
cubicweb.web.views. That is: the Yams schema is introspected to determine
how properties should be generated and any additionnal control over this can
be performed through uicfg declarations [1].
To illustrate let's consider the following schema definitions which:
class Book(EntityType):
title = String(required=True)
publication_date = Datetime(required=True)
class Illustration(EntityType):
data = Bytes(required=True)
class illustrates(RelationDefinition):
subject = 'Illustration'
object = 'Book'
cardinality = '1*'
composite = 'object'
inlined = True
class Author(EntityType):
name = String(required=True)
class author(RelationDefinition):
subject = 'Book'
object = 'Author'
cardinality = '1*'
class Topic(EntityType):
name = String(required=True)
class topics(RelationDefinition):
subject = 'Book'
object = 'Topic'
cardinality = '**'
and consider, as before, JSON Schema documents in different contexts for the
the Book entity type:
in view context:
{
"$ref": "#/definitions/Book",
"definitions": {
"Book": {
"additionalProperties": false,
"properties": {
"author": {
"items": {
"type": "string"
},
"title": "author",
"type": "array"
},
"publication_date": {
"format": "date-time",
"title": "publication_date",
"type": "string"
},
"title": {
"title": "title",
"type": "string"
},
"topics": {
"items": {
"type": "string"
},
"title": "topics",
"type": "array"
}
},
"title": "Book",
"type": "object"
}
}
}
We have a single Book definition in this document, in which we find
attributes defined in the Yams schema (title and publication_date). We
also find the two relations where Book is involved: topics and
author, both appearing as a single array of "string" items. The author
relationship appears like that because it is mandatory but not composite.
On the other hand, the topics relationship has the following uicfg rule:
uicfg.primaryview_section.tag_subject_of(('Book', 'topics', '*'), 'attributes')
so that it's definition appears embedded in the document of Book
definition.
A typical JSON representation of a Book entity would be:
{
"author": [
"Ernest Miller Hemingway"
],
"title": "The Old Man and the Sea",
"topics": [
"sword fish",
"cuba"
]
}
in creation context:
{
"$ref": "#/definitions/Book",
"definitions": {
"Book": {
"additionalProperties": false,
"properties": {
"author": {
"items": {
"oneOf": [
{
"enum": [
"855"
],
"title": "Ernest Miller Hemingway"
},
{
"enum": [
"857"
],
"title": "Victor Hugo"
}
],
"type": "string"
},
"maxItems": 1,
"minItems": 1,
"title": "author",
"type": "array"
},
"publication_date": {
"format": "date-time",
"title": "publication_date",
"type": "string"
},
"title": {
"title": "title",
"type": "string"
}
},
"required": [
"title",
"publication_date"
],
"title": "Book",
"type": "object"
}
}
}
notice the differences, we now only have attributes and required relationships
(author) in this schema and we have the required listing mandatory
attributes; the author property is represented as an array which items
consist of pre-existing objects of the author relationship (namely
Author entities).
Now assume we add the following uicfg declaration:
uicfg.autoform_section.tag_object_of(('*', 'illustrates', 'Book'), 'main', 'inlined')
the JSON Schema for creation context will be:
{
"$ref": "#/definitions/Book",
"definitions": {
"Book": {
"additionalProperties": false,
"properties": {
"author": {
"items": {
"oneOf": [
{
"enum": [
"855"
],
"title": "Ernest Miller Hemingway"
},
{
"enum": [
"857"
],
"title": "Victor Hugo"
}
],
"type": "string"
},
"maxItems": 1,
"minItems": 1,
"title": "author",
"type": "array"
},
"illustrates": {
"items": {
"$ref": "#/definitions/Illustration"
},
"title": "illustrates_object",
"type": "array"
},
"publication_date": {
"format": "date-time",
"title": "publication_date",
"type": "string"
},
"title": {
"title": "title",
"type": "string"
}
},
"required": [
"title",
"publication_date"
],
"title": "Book",
"type": "object"
},
"Illustration": {
"additionalProperties": false,
"properties": {
"data": {
"format": "data-url",
"title": "data",
"type": "string"
}
},
"required": [
"data"
],
"title": "Illustration",
"type": "object"
}
}
}
We now have an additional illustrates property modelled as an array of
#/definitions/Illustration, the later also added the the document as
an additional definition entry.
Conclusion
This post illustrated how a basic (CRUD) HTTP API based on JSON Schema could
be build for a CubicWeb application using cubicweb-jsonschema. We have seen
a couple of details on JSON Schema generation and how it can be controlled.
Feel free to comment and provide feedback on this feature set as well as
open the discussion with more use cases.
Next time, we'll discuss how hypermedia controls can be added the HTTP API
that cubicweb-jsonschema provides.
[1]
this choice is essentially driven by simplicity and conformance
when the existing behavior to help migration of existing applications.
[Less]
|
Posted
over 7 years
ago
by
Arthur Lutz
Following the "release often, release early" mantra, I thought it
might be a good idea to apply it to monitoring on one of our client
projects. So right from the demo stage where we deliver a new version
every few weeks (and sometimes every few
... [More]
days), we setup some
monitoring.
Monitoring performance
The project is an application built with the CubicWeb platform, with
some ElasticSearch for indexing and searching. As with any complex
stack, there are a great number of places where one could monitor
performance metrics.
Here are a few things we have decided to monitor, and with what tools.
Monitoring CubicWeb
To monitor our running Python code, we have decided to use statsd, since it is already built into
CubicWeb's core. Out of the box, you can configure a
statsd server address in your all-in-one.conf configuration. That will
send out some timing statistics about some core functions.
The statsd server (there a numerous implementations, we use a simple
one : python-pystatsd) gets the raw metrics and outputs
them to carbon which
stores the time series data in whisper files (which can be
swapped out for a different technology if need be).
If we are curious about a particular function or view that might be
taking too long to generate or slow down the user experience, we can
just add the @statsd_timeit
decorator there. Done. It's monitored.
statsd monitoring is a fire-and-forget UDP type of monitoring, it
should not have any impact on the performance of what you are
monitoring.
Monitoring Apache
Simply enough we re-use the statsd approach by plugging in an apache
module to time the HTTP responses sent back by apache. With nginx
and varnish, this is also really easy.
One of the nice things about this part is that we can then get
graphs of errors since we will differentiate OK 200 type codes from
500 type codes (HTTP codes).
Monitoring ElasticSearch
ElasticSearch comes with some metrics in GET /_stats endpoint, the
same goes for individual nodes, individual indices and even at cluster
level. Some popular tools can be installed through the ElasticSearch
plugin system or with Kibana (plugin system there too).
We decided on a different approach that fitted well with our other
tools (and demonstrates their flexibility!) : pull stats out of
ElasticSearch with SaltStack,
push them to Carbon, pull them out with Graphite and display
them in Grafana (next to our other metrics).
On the SaltStack side, we wrote a two line execution module (elasticsearch.py)
import requests
def stats:
return request.get('http://localhost:9200/_stats').json()
This gets shipped using the custom execution modules mechanism
(_modules and saltutils.sync_modules), and is executed every minute
(or less) in the salt scheduler. The
resulting dictionary is fed to the carbon returner that is configured
to talk to a carbon server somewhere nearby.
# salt demohost elasticsearch.stats
[snip]
{ "indextime_inmillis" : 30,
[snip]
Monitoring web metrics
To evaluate parts of the performance of a web page we can look at some
metrics such as the number of assets the browser will need to
download, the size of the assets (js, css, images, etc) and even
things such as the number of subdomains used to deliver assets. You
can take a look at such metrics in most developer tools available in
the browser, but we want to graph this over time. A nice tool for this
is sitespeed.io (written in javascript
with phantomjs). Out of the box, it has a graphite outputter so
we just have to add --graphiteHost FQDN. sitespeed.io even
recommends using grafana to visualize the
results and publishes some example dashboards that can be adapted to
your needs.
The sitespeed.io command is configured and run by salt using pillars
and its scheduler.
We will have to take a look at using their jenkins plugin with our
jenkins continuous integration instance.
Monitoring crashes / errors / bugs
Applications will have bugs (in particular when released often to get
a client to validate some design choices early). Level 0 is having
your client calling you up saying the application has crashed. The
next level is watching some log somewhere to see those errors pop
up. The next level is centralised logs on which you can monitor the
numerous pieces of your application (rsyslog over UDP helps here,
graylog might be a good solution for
visualisation).
When it starts getting useful and usable is when your bugs get
reported with some rich context. That's when using sentry gets in. It's free software developed on github (although the website does not
really show that) and it is written in python, so it was a good match
for our culture. And it is pretty awesome too.
We plug sentry into our WSGI pipeline (thanks to cubicweb-pyramid) by installing
and configuring the sentry cube : cubicweb-sentry. This will catch
rich context bugs and provide us with vital information about what the user
was doing when the crash occured.
This also helps sharing bug information within a team.
The sentry cube reports on errors being raised when using the web
application, but can also catch some errors when running some
maintenance or import commands (ccplugins in CubicWeb). In this
particular case, a lot of importing is being done and Sentry can
detect and help us triage the import errors with context on which
files are failing.
Monitoring usage / client side
This part is a bit neglected for the moment. Client side we can use
Javascript to monitor usage. Some basic metrics can come from piwik which is usually used for audience
statistics. To get more precise statistics we've been told Boomerang has an interesting
approach, enabling a closer look at how fast a page was displayed
client side, how much time was spend on DNS, etc.
On the client side, we're also looking at two features of the Sentry
project : the raven-js
client which reports Javascript errors directly from the browser to
the Sentry server, and the user feedback form which captures some
context when something goes wrong or a user/client wants to report
that something should be changed on a given page.
Load testing - coverage
To wrap up, we also often generate traffic to catch some bugs and
performance metrics automatically :
wget --mirror $URL
linkchecker $URL
for $search_term in cat corpus; do wget URL/$search_term ; done
wapiti $URL --scope page
nikto $URL
Then watch the graphs and the errors in Sentry... Fix them. Restart.
Graphing it in Grafana
We've spend little time on the dashboard yet since we're concentrating on collecting the metrics for now. But here is a glimpse of the "work in progress" dashboard which combines various data sources and various metrics on the same screen and the same time scale.
Further plans
internal health checks, we're taking a look at python-hospital and healthz: Stop
reverse engineering applications and start monitoring from the
inside (Monitorama) (the idea is to
distinguish between the app is running and the app is serving it's
purpose), and pyramid_health
graph the number of Sentry errors and the number of types of errors:
the sentry API should be able to give us this information. Feed it to
Salt and Carbon.
setup some alerting : next versions of Grafana will be doing that, or with elastalert
setup "release version X" events in Graphite that are displayed in
Grafana, maybe with some manual command or a postcreate command when
using docker-compose up ?
make it easier for devs to have this kind of setup. Using this suite
of tools in developement might sometimes be overkill, but can be
useful.
[Less]
|
Posted
over 7 years
ago
by
Arthur Lutz
Following the "release often, release early" mantra, I thought it
might be a good idea to apply it to monitoring on one of our client
projects. So right from the demo stage where we deliver a new version
every few weeks (and sometimes every few
... [More]
days), we setup some
monitoring.
Monitoring performance
The project is an application built with the CubicWeb platform, with
some ElasticSearch for indexing and searching. As with any complex
stack, there are a great number of places where one could monitor
performance metrics.
Here are a few things we have decided to monitor, and with what tools.
Monitoring CubicWeb
To monitor our running Python code, we have decided to use statsd, since it is already built into
CubicWeb's core. Out of the box, you can configure a
statsd server address in your all-in-one.conf configuration. That will
send out some timing statistics about some core functions.
The statsd server (there a numerous implementations, we use a simple
one : python-pystatsd) gets the raw metrics and outputs
them to carbon which
stores the time series data in whisper files (which can be
swapped out for a different technology if need be).
If we are curious about a particular function or view that might be
taking too long to generate or slow down the user experience, we can
just add the @statsd_timeit
decorator there. Done. It's monitored.
statsd monitoring is a fire-and-forget UDP type of monitoring, it
should not have any impact on the performance of what you are
monitoring.
Monitoring Apache
Simply enough we re-use the statsd approach by plugging in an apache
module to time the HTTP responses sent back by apache. With nginx
and varnish, this is also really easy.
One of the nice things about this part is that we can then get
graphs of errors since we will differentiate OK 200 type codes from
500 type codes (HTTP codes).
Monitoring ElasticSearch
ElasticSearch comes with some metrics in GET /_stats endpoint, the
same goes for individual nodes, individual indices and even at cluster
level. Some popular tools can be installed through the ElasticSearch
plugin system or with Kibana (plugin system there too).
We decided on a different approach that fitted well with our other
tools (and demonstrates their flexibility!) : pull stats out of
ElasticSearch with SaltStack,
push them to Carbon, pull them out with Graphite and display
them in Grafana (next to our other metrics).
On the SaltStack side, we wrote a two line execution module (elasticsearch.py)
import requests
def stats():
return request.get('http://localhost:9200/_stats').json()
This gets shipped using the custom execution modules mechanism
(_modules and saltutils.sync_modules), and is executed every minute
(or less) in the salt scheduler. The
resulting dictionary is fed to the carbon returner that is configured
to talk to a carbon server somewhere nearby.
# salt demohost elasticsearch.stats
[snip]
{ "indextime_inmillis" : 30,
[snip]
Monitoring web metrics
To evaluate parts of the performance of a web page we can look at some
metrics such as the number of assets the browser will need to
download, the size of the assets (js, css, images, etc) and even
things such as the number of subdomains used to deliver assets. You
can take a look at such metrics in most developer tools available in
the browser, but we want to graph this over time. A nice tool for this
is sitespeed.io (written in javascript
with phantomjs). Out of the box, it has a graphite outputter so
we just have to add --graphiteHost FQDN. sitespeed.io even
recommends using grafana to visualize the
results and publishes some example dashboards that can be adapted to
your needs.
The sitespeed.io command is configured and run by salt using pillars
and its scheduler.
We will have to take a look at using their jenkins plugin with our
jenkins continuous integration instance.
Monitoring crashes / errors / bugs
Applications will have bugs (in particular when released often to get
a client to validate some design choices early). Level 0 is having
your client calling you up saying the application has crashed. The
next level is watching some log somewhere to see those errors pop
up. The next level is centralised logs on which you can monitor the
numerous pieces of your application (rsyslog over UDP helps here,
graylog might be a good solution for
visualisation).
When it starts getting useful and usable is when your bugs get
reported with some rich context. That's when using sentry gets in. It's free software developed on github (although the website does not
really show that) and it is written in python, so it was a good match
for our culture. And it is pretty awesome too.
We plug sentry into our WSGI pipeline (thanks to cubicweb-pyramid) by installing
and configuring the sentry cube : cubicweb-sentry. This will catch
rich context bugs and provide us with vital information about what the user
was doing when the crash occured.
This also helps sharing bug information within a team.
The sentry cube reports on errors being raised when using the web
application, but can also catch some errors when running some
maintenance or import commands (ccplugins in CubicWeb). In this
particular case, a lot of importing is being done and Sentry can
detect and help us triage the import errors with context on which
files are failing.
Monitoring usage / client side
This part is a bit neglected for the moment. Client side we can use
Javascript to monitor usage. Some basic metrics can come from piwik which is usually used for audience
statistics. To get more precise statistics we've been told Boomerang has an interesting
approach, enabling a closer look at how fast a page was displayed
client side, how much time was spend on DNS, etc.
On the client side, we're also looking at two features of the Sentry
project : the raven-js
client which reports Javascript errors directly from the browser to
the Sentry server, and the user feedback form which captures some
context when something goes wrong or a user/client wants to report
that something should be changed on a given page.
Load testing - coverage
To wrap up, we also often generate traffic to catch some bugs and
performance metrics automatically :
wget --mirror $URL
linkchecker $URL
for $search_term in cat corpus; do wget URL/$search_term ; done
wapiti $URL --scope page
nikto $URL
Then watch the graphs and the errors in Sentry... Fix them. Restart.
Graphing it in Grafana
We've spend little time on the dashboard yet since we're concentrating on collecting the metrics for now. But here is a glimpse of the "work in progress" dashboard which combines various data sources and various metrics on the same screen and the same time scale.
Further plans
internal health checks, we're taking a look at python-hospital and healthz: Stop
reverse engineering applications and start monitoring from the
inside (Monitorama) (the idea is to
distinguish between the app is running and the app is serving it's
purpose), and pyramid_health
graph the number of Sentry errors and the number of types of errors:
the sentry API should be able to give us this information. Feed it to
Salt and Carbon.
setup some alerting : next versions of Grafana will be doing that, or with elastalert
setup "release version X" events in Graphite that are displayed in
Grafana, maybe with some manual command or a postcreate command when
using docker-compose up ?
make it easier for devs to have this kind of setup. Using this suite
of tools in developement might sometimes be overkill, but can be
useful.
[Less]
|
Posted
about 8 years
ago
by
Sylvain Thenault
Some of us at Logilab made some kind of retrospective of the processes surrounding CubicWeb (core) development. One of the important point is that we decided that Logilab would not resume the bi-monthly meetings with other developers and users to
... [More]
discuss and update the roadmap.
Instead, we will do more Blog Driven Development (as started by Denis on this blog) to keep the community up to date with our internal effort. We will continue to use the mailing-list to discuss with everybody what we contribute to CubicWeb.
We will also try to organize a sprint in the coming months, but we are not able to commit on a date for now, because we are under a heavy load with customer work (which is a good thing for us and CubicWeb, of course).
Another topic was to set up a kind of working agreement for those developing on the core. For a start, we agreed on the following points:
we stop the two-steps review process
integrators commit to handling pending reviews under a 7 days time frame
an integrator is not allowed to integrate his or her own patches
incoming patches should not make the QA numbers go down, and should be py3k compatible.
QA numbers are still to be defined in a forthcoming internal sprint on continuous integration system. Current integrators are Julien, David, Denis, Florent and Sylvain. If you're interested, let us know on the mailing-list. [Less]
|
Posted
about 8 years
ago
by
Sylvain Thenault
Some of us at Logilab made some kind of retrospective of the processes surrounding CubicWeb (core) development. One of the important point is that we decided that Logilab would not resume the bi-monthly meetings with other developers and users to
... [More]
discuss and update the roadmap.
Instead, we will do more Blog Driven Development (as started by Denis on this blog) to keep the community up to date with our internal effort. We will continue to use the mailing-list to discuss with everybody what we contribute to CubicWeb.
We will also try to organize a sprint in the coming months, but we are not able to commit on a date for now, because we are under a heavy load with customer work (which is a good thing for us and CubicWeb, of course).
Another topic was to set up a kind of working agreement for those developing on the core. For a start, we agreed on the following points:
we stop the two-steps review process
integrators commit to handling pending reviews under a 7 days time frame
an integrator is not allowed to integrate his or her own patches
incoming patches should not make the QA numbers go down, and should be py3k compatible.
QA numbers are still to be defined in a forthcoming internal sprint on continuous integration system. Current integrators are Julien, David, Denis, Florent and Sylvain. If you're interested, let us know on the mailing-list. [Less]
|
Posted
about 8 years
ago
by
Denis Laxalde
Following the introduction post about rethinking the web user
interface of CubicWeb, this article will address the topic of the Web API to
exchange data between the client and the server. As mentioned earlier, this
question is somehow central and
... [More]
deserves particular interest, and better early
than late. Of the two candidate representations previously identified
Hydra and JSON API, this article will focus on the later.
Hopefully, this will give a better insight of the capabilities and limits of
this specification and would help take a decision, though a similar experiment
with another candidate would be good to have. Still in the process of blog
driven development, this post has several open questions from which a
discussion would hopefully emerge...
A glance at JSON API
JSON API is a specification for building APIs that use JSON as a
data exchange format between clients and a server. The media type is
application/vnd.api+json. It has a 1.0 version available from mid-2015.
The format has interesting features such as the ability to build
compound documents (i.e. response made of several, usually
related, resources) or to specify filtering, sorting and pagination.
A document following the JSON API format basically represents resource
objects, their attributes and relationships as well as some links
also related to the data of primary concern.
Taking the example of a Ticket resource modeled after the tracker cube,
we could have a JSON API document formatted as:
GET /ticket/987654
Accept: application/vnd.api+json
{
"links": {
"self": "https://www.cubicweb.org/ticket/987654"
},
"data": {
"type": "ticket",
"id": "987654",
"attributes": {
"title": "Let's use JSON API in CubicWeb"
"description": "Well, let's try, at least...",
},
"relationships": {
"concerns": {
"links": {
"self": "https://www.cubicweb.org/ticket/987654/relationships/concerns",
"related": "https://www.cubicweb.org/ticket/987654/concerns"
},
"data": {"type": "project", "id": "1095"}
},
"done_in": {
"links": {
"self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
"related": "https://www.cubicweb.org/ticket/987654/done_in"
},
"data": {"type": "version", "id": "998877"}
}
}
},
"included": [{
"type": "project",
"id": "1095",
"attributes": {
"name": "CubicWeb"
},
"links": {
"self": "https://www.cubicweb.org/project/cubicweb"
}
}]
}
In this JSON API document, top-level members are links, data
and included. The later is here used to ship some resources (here a
"project") related to the "primary data" (a "ticket") through the "concerns"
relationship as denoted in the relationships object (more on this later).
While the decision of including or not these related resources along with the
primary data is left to the API designer, JSON API also offers a specification
to build queries for inclusion of related resources. For example:
GET /ticket/987654?include=done_in
Accept: application/vnd.api+json
would lead to a response including the full version resource along with the
above content.
Enough for the JSON API overview. Next I'll present how various aspects of
data fetching and modification can be achieved through the use of JSON API in
the context of a CubicWeb application.
CRUD
CRUD of resources is handled in a fairly standard way in JSON API, relying
of HTTP protocol semantics.
For instance, creating a ticket could be done as:
POST /ticket
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json
{
"data": {
"type": "ticket",
"attributes": {
"title": "Let's use JSON API in CubicWeb"
"description": "Well, let's try, at least...",
},
"relationships": {
"concerns": {
"data": { "type": "project", "id": "1095" }
}
}
}
}
Then updating it (assuming we got its id from a response to the above
request):
PATCH /ticket/987654
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json
{
"data": {
"type": "ticket",
"id": "987654",
"attributes": {
"description": "We'll succeed, for sure!",
},
}
}
Relationships
In JSON API, a relationship is in fact a first class resource as it is defined
by a noun and an URI through a link object. In this respect, the
client just receives a couple of links and can eventually operate on them
using the proper HTTP verb. Fetching or updating relationships is done using
the special <resource url>/relationships/<relation type> endpoint (self
member of relationships items in the first example). Quite naturally, the
specification relies on GET verb for fetching targets, PATCH for
(re)setting a relation (i.e. replacing its targets), POST for adding targets
and DELETE to drop them.
GET /ticket/987654/relationships/concerns
Accept: application/vnd.api+json
{
"data": {
"type": "project",
"id": "1095"
}
}
PATCH /ticket/987654/relationships/done_in
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json
{
"data": {
"type": "version",
"id": "998877"
}
}
The body of request and response of this <resource
url>/relationships/<relation type> endpoint consists of so-called resource
identifier objects which are lightweight representation of resources
usually only containing information about their "type" and "id" (enough to
uniquely identify them).
Related resources
Remember the related member appearing in relationships links in the first
example?
[ ... ]
"done_in": {
"links": {
"self": "https://www.cubicweb.org/ticket/987654/relationships/done_in",
"related": "https://www.cubicweb.org/ticket/987654/done_in"
},
"data": {"type": "version", "id": "998877"}
}
[ ... ]
While this is not a mandatory part of the specification, it has an interesting
usage for fetching relationship targets. In contrast with the
.../relationships/... endpoint, this one is expected to return plain
resource objects (which attributes and relationships information in
particular).
GET /ticket/987654/done_in
Accept: application/vnd.api+json
{
"links": {
"self": "https://www.cubicweb.org/998877"
},
"data": {
"type": "version",
"id": "998877",
"attributes": {
"number": 4.2
},
"relationships": {
"version_of": {
"self": "https://www.cubicweb.org/998877/relationships/version_of",
"data": { "type": "project", "id": "1095" }
}
}
},
"included": [{
"type": "project",
"id": "1095",
"attributes": {
"name": "CubicWeb"
},
"links": {
"self": "https://www.cubicweb.org/project/cubicweb"
}
}]
}
Meta information
The JSON API specification allows to include non-standard information using a
so-called meta object. This can be found in various place of the document
(top-level, resource objects or relationships object). Usages of this
field is completely free (and optional). For instance, we could use this field
to store the workflow state of a ticket:
{
"data": {
"type": "ticket",
"id": "987654",
"attributes": {
"title": "Let's use JSON API in CubicWeb"
},
"meta": { "state": "open" }
}
Permissions
Permissions are part of metadata to be exchanged during request/response
cycles. As such, the best place to convey this information is probably within
the headers. According to JSON API's FAQ, this is also the recommended
way for a resource to advertise on supported actions.
So for instance, response to a GET request could include Allow headers,
indicating which request methods are allowed on the primary resource
requested:
GET /ticket/987654
Allow: GET, PATCH, DELETE
An HEAD request could also be used for querying allowed actions on links
(such as relationships):
HEAD /ticket/987654/relationships/comments
Allow: POST
This approach has the advantage of being standard HTTP, no particular
knowledge of the permissions model is required and the response body is not
cluttered with these metadata.
Another possibility would be to rely use the meta member of JSON API data.
{
"data": {
"type": "ticket",
"id": "987654",
"attributes": {
"title": "Let's use JSON API in CubicWeb"
},
"meta": {
"permissions": ["read", "update"]
}
}
}
Clearly, this would minimize the amount client/server requests.
More Hypermedia controls
With the example implementation described above, it appears already possible to
manipulate several aspects of the entity-relationship database following a
CubicWeb schema: resources fetching, CRUD operations on entities, set/delete
operations on relationships. All these "standard" operations are discoverable
by the client simply because they are baked into the JSON API format: for
instance, adding a target to some relationship is possible by POSTing to the
corresponding relationship resource something that
conforms to the schema.
So, implicitly, this already gives us a fairly good level of Hypermedia
control so that we're not so far from having a mature REST architecture
according to the Richardson Maturity Model. But beyond these "standard"
discoverable actions, the JSON API specification does not address yet
Hypermedia controls in a generic manner (see this interesting
discussion about extending the specification for this
purpose).
So the question is: would we want more? Or, in other words, do we need to
define "actions" which would not map directly to a concept in the application
model?
In the case of a CubicWeb application, the most obvious example (that I could
think of) of where such an "action" would be needed is workflow state
handling. Roughly, workflows in CubicWeb are modeled through two entity
types State and TrInfo (for "transition information"), the former being
handled through the latter, and a relationship in_state between the
workflowable entity type at stake and its current State. It does not appear
so clearly how would one model this in terms of HTTP resource. (Arguably we
wouldn't want to expose the complexity of Workflow/TrInfo/State data model to
the client, nor can we simply expose this in_state relationship, as a client
would not be able to simply change the state of a entity by updating the
relation). So what would be a custom "action" to handle the state of a
workflowable resource? Back in our tracker example, how would we advertise to
the client the possibility to perform "open"/"close"/"reject" actions on a
ticket resource? Open question...
Request for comments
In this post, I tried to give an overview of a possible usage of JSON
API to build a Web API for CubicWeb. Several aspects were discussed
from simple CRUD operations, to relationships handling or non-standard
actions. In many cases, there are open questions for which I'd love to receive
feedback from the community.
Recalling that this topic is a central part of the experiment towards building
a client-side user interface to CubicWeb, the more discussion it gets, the
better!
For those wanting to try and play themselves with the experiments, have a look
at the code. This is a
work-in-progress/experimental implementation, relying on Pyramid for content
negotiation and route traversals.
What's next? Maybe an alternative experiment relying on Hydra? Or an
orthogonal one playing with the schema client-side? [Less]
|