Drizzle

Last week Henrik announced that he was stepping down from Drizzle development to pursue work on Solon Voting.  I must considerately step down too, for after many long months and late nights, my startup has started up: Test Noir.  Between working full-time at Percona and Test Noir I have zero time for any other coding.  It [...]

The 4th release of the GreenLake series is out. Major changes include: * Clang support by Brian Aker * Sysbench Qewpie testing suite by Sharan and Patrick Crews * Numerous bug fixes by coldtobi * Dynamic plugin support by Anshu Kumar * Improved ... [More] Documentation On a side note, a proof of concept was done for Drizzle as a backend for OpenStack during this release as well. [Less]

The 4th release of the GreenLake series is out. Major changes include: Clang support by Brian Aker Sysbench Qewpie testing suite by Sharan and Patrick Crews Numerous bug fixes by coldtobi Dynamic plugin support by Anshu Kumar Improved Documentation ... [More] On a side note, a proof of concept was done for Drizzle as a backend for OpenStack during this release as well. [Less]

Yesterday, over at my personal blog, I blogged about the impact of the MySQL slow query log. Since we’re working on Percona Server 5.6, I did wonder if this was a good opportunity to re-examine how we could provide slow query log type functionality ... [More] to our users. The slow query log code inside the MySQL server does several things that aren’t good for concurrency on modern systems. For starters, it takes a mutex while both doing formatting of what to write and while doing the write(2) system call itself. This is, in fact, entirely unneccesary. From the man page for the write system call: If the file was open(2)ed with O_APPEND, the file offset is first set to the end of the file before writing. The adjustment of the file offset and the write operation are performed as an atomic step. So we can pretty easily see that for our case of writing the slow query log, if we had a buffer that was the log entry, we could just call write(2) and never take a mutex. We trade some memory usage (for the buffer) for concurrency – pushing all the concurrency problems down into the operating system. The main challenge with this strategy is log file rotation, as if you just close(2) a file descriptor that others may be write(2)ing to then bad things may happen (namely that the file descriptor gets reused for something else.. say an innodb data file, and writing part of your slow query log as an innodb page will not end in hugs and puppies). So, in order to test my theory, I sat down and wrote an audit plugin. It turns out I had to extend the MySQL plugin API to be able to reproduce all the information in the slow query log. The code for my prototype can be found at lp:~stewart/percona-server/5.6-slow-query-plugin-prototype (this revision) – this will probably build fine against 5.5 too, but I haven’t tested. To build it you’ll need the Intel Threading Building Blocks headers installed as I used their tbb::atomic<> implementation. My solution to log file rotation is to basically count how many people may be using it and waiting until this drops to zero before closing the file descriptor. This avoids taking any locks around calling write(), which is the most time consuming part of writing the log. I’m not convinced if this is the best solution or not, but it appears to work and I think it’s race free. No improvement like this is complete without benchmarks. My benchmarks (“mysqlslap -q ‘select 1;’ –number-of-queries=1000000 –concurrency=64″ on my laptop) come out at the following: straight MySQL (no slow query log): 13 seconds MySQL with slow query log enabled: 18 seconds My audit plugin slow query log: 15 seconds straight Drizzle (no slow query log): 8 seconds So that’s a quite promising start. [Less]

So, what impact does enabling the slow query log have on MySQL? I decided to run some numbers. I’m using my laptop, as we all know the currently most-deployed database servers have mulitple cores, SSDs and many GB of RAM. For the curious: Intel(R) ... [More] Core(TM) i7-2620M CPU @ 2.70GHz The benchmark is going to be: mysqlslap -u root test -S var/tmp/mysqld.1.sock -q 'select 1;' --number-of-queries=1000000 --concurrency=64 --create-schema=test Which is pretty much “run a whole bunch of nothing, excluding all the overhead of storage engines, optimizer… and focus on logging”. My first run was going to be with the slow query log on. I’ll start the server with mysql-test-run.pl as it’s just easy: eatmydata ./mysql-test-run.pl --start-and-exit --mysqld=--slow-query-log --mysqld=--long-query-time=0 The results? It took 18 seconds. How long without the slow query log (starting with mysql-test-run.pl again, but this time without any of the extra mysqld options)? 13 seconds. How does this compare to a Drizzle baseline? On a freshly build Drizzle trunk, using the same mysqlslap binary I used above, connecting via UNIX socket: 8 seconds. [Less]

We are glad to announce the availability of Percona Playback 0.4, the first release where we’ve released binary packages alongside our source release. Binary packages for RHEL/CentOS 6 are available with Debian/Ubuntu packages soon to come (we are ... [More] ensuring our dependencies are correct before publishing). Packages and source may be downloaded from here and are available in our EXPERIMENTAL repositories. Changes in this release (since 0.3): Produce binary packages for CentOS/RHEL and Debian/Ubuntu. Initial libdrizzle based database client (can be used instead of libmysqlclient). Fix #1021577, improve accurate mode of tcpdump playback. We now build on CentOS/RHEL 5 (with appropriate libraries) Documentation updates [Less]

As a part of Google Summer of Code 2012, I worked on Improved Performance Regression Monitoring for DrizzleDB.. At the bottom, its a Quality Assurance and Automated Testing project.. Drizzle already has *Drizzle Automation* which deals with ... [More] automating QA processes, code coverage analysis and benchmarking.. However due to the following issues, its not preferred much.. 1. Its not in-tree 2. Its more complex 3. Its less user friendly In order to nullify this, the kewpie tests came into effect. Kewpie is a test-runner for MySQL based databases. Its cleaner, simpler and very much effective. Moreover, its included in-tree.. The project’s intent is to import sysbench tests from drizzle automation to kewpie. SysBench is a benchmark tool. It has benchmarks for various system parameters, which includes an oltp benchmark ( for testing database servers ). Following are the enhancements made into the Drizzle Trunk : 1. Added options for kewpie: Options for dsn-string and mailing test reports have been included. The dsn-string specified via the –results-db-dsn option is used for connecting with a database. And if the user needs to send the report of a test to a mail id, it can be done easily be specifying the mail id with the –email-report-tgt option. You can have a deeper look over here. 2. Imported sysbench tests: There are two variants of sysbench tests. One is a *readonly* test which shoots the database with SELECT queries and the other is *readwrite* test which includes INSERT, UPDATE, DELETE and DROP queries too.. These tests are imported to kewpie in an  ingenious way.. Instead of having the entire test code in each test case, there is a base class *sysbenchTestCase.py* in /tests/lib/util. Both the test cases, readonly and readwrite ( /tests/qp_tests/sysbench ), inherits this base class. Further, these test cases have only the configuration options in them, which is unique for both.. Peep here for more details. 3. Improved documetation: Documentations for kewpie and sysbench were modified, and documentation for drizzletest commands was included. The latter one is not yet complete, however the api calls can be glanced at, in the documentation.. More information on each will be documented very soon.. [Less]

The desire to contribute to opensource sprouted in me quite a long time ago. Browsing over the web and learning more about it, learning more from my friends, the eagerness reached a peak. It was at this point that I came to know about Google Summer ... [More] of Code. To put it in a nutshell, Google Summer of Code ( GSoC ) is a program hosted by Google, where a lot of opensource organizations participate. Students can suggest their project ideas or develop existing project ideas, work on and develop their module(s) and receive stipend from Google. I dived into Drizzle and chose a project on Improved Performance Regression Monitoring, which is related to Quality Assurance ( QA ). My project got selected and henceforth I need to start coding for my module. With the support from my mentor, Drizzle developers and a friend of mine, I learnt the basics. I also did some work on learning python, as I had to code my module in python and I was new to that language. The three months time working with Google and Drizzle was really a groovy experience for me. I learnt a lot, enjoyed my work, and made friends and so..on. And at the close of the program, I had my modules completed, generated good documentations and all set. My code was merged into the Drizzle Trunk  ( the main source code ) . And that’s the most awaited moment!! I have become a Developer ( contributor ) at Drizzle!! A dream came true : ) [Less]

Here is a piece of code that illustrates how to create a Drizzle object, a connection object, establish a connection, execute a query, buffer its result and do some basic operations on it. The code is pretty straight forward. DRIZZLE_RETURN_OK ... [More] indicates the successful execution of a function. Make sure that you have libdrizzle installed in your machine. #include <libdrizzle/libdrizzle.h> #include <stdio.h> #include <stdlib.h> int main() { drizzle_st *drizzle; drizzle_con_st *con; if ((drizzle= drizzle_create(NULL)) == NULL) { printf(“Error creating drizzle object\n”); return -1; } con= drizzle_con_add_tcp(drizzle, NULL, “localhost”, 4427, “root”, “password”, “db”, DRIZZLE_NON_BLOCKING); drizzle_return_t ret= drizzle_con_connect(con); if (ret == DRIZZLE_RETURN_OK) { printf(“Connection Established\n”); } drizzle_result_st *result; int count= 0; result= drizzle_query(con, result, “SHOW DATABASES”, strlen(“SHOW DATABASES”), &ret); ret= drizzle_result_buffer(result); printf(“%d\n”, (int)drizzle_result_row_count(result)); drizzle_con_free(con); drizzle_free(drizzle); return 0; } To compile this program, one needs to link to libdrizzle. Use the following command to compile the code file. (Assuming that the code file is connect.c and that you installed libdrizzle-1.0 headers at /usr/include) gcc -o connect connect.c -ldrizzle -I /usr/include/libdrizzle-1.0 -L /usr/lib Hope this helps. Happy hacking folks! [Less]

This year was my third consecutive Google Summer of Code and I’ve done all three of my GSoCs for the Drizzle community. The first time I worked on refactoring the commandline options processing system using boost::program_options. The second time I ... [More] worked on a stored procedure interface for Drizzle. This year I got a very interesting project to work on. My mentor, Brian Aker, asked me if I could work on adding Drizzle backend support for OpenStack. I was very thrilled since I’ve been long trying to contribute to the OpenStack community and OpenStack is a community that I’ve admired since its infant days. So, I happily said yes. Brian charted out a plan asking me to try adding a plugin for Collectd which could be used to collect the Drizzle server’s statistics from time to time. As part of this I had to work with libdrizzle. There were not many concrete examples of libdrizzle usage. I somehow managed to write a simple “Hello,World” kind of libdrizzle based C program which I had used as a base to start working on the plugin. But, because of all the relocating and stuff that was happening in my personal life I never got to finish the plugin and it is still left in a half baked state. I will add a blog showing the basic program that I had written using libdrizzle so that people could use it as a base to build on. I then moved on to modifying devstack to script so that it uses Drizzle as a backend instead of MySQL. Monty Taylor had already told me that all the Drizzle support is already built in as a part of SQLAlchemy and I wouldn’t have to do much. Hence I started to remove all the MySQL related code and adding its equivalent Drizzle code. Diving further in I had noticed that I would have to modify some of the SQLAlchemy library code and some of the code in Nova which I did and got working OpenStack with Drizzle. I will add another blog post covering how I had achieved the backend support as well. All in all, I had fun working on this project this year and the amount of satisfaction that I got on seeing OpenStack run with Drizzle in the backend is priceless. Ultimately I too have contributed to OpenStack in one form. Now, Drizzle is truely “on the cloud”. [Less]