Turnkey Linux

Appliance:  Ruby on Rails Appliance Changes: Replaced Ruby with Ruby Enterprise (performance). Set MySQL root password on firstboot (convenience, security). Force MySQL to use ... [More] Unicode/UTF8. Set postfix MTA myhostname to localhost (bugfix). Included libreadline-dev so rails console works (bugfix). Major component versions ruby-enterprise 1.8.7-2010.02_i386_ubuntu10.04 mysql-server 5.1.41-3ubuntu12.8 apache2 2.2.14-5ubuntu8.4 build-essential 11.4build1 Note: Please refer to turnkey-core's changelog for changes common to all appliances. Here we only describe changes specific to this appliance. Links Release meta-files (signature, manifest) [Less]

Appliance:  ProjectPier Appliance Changes: ProjectPier: Installed and configured kampPro theme (similar to 37signals basecamp). Removed InnoDB logfile workaround (bad idea ... [More] for a transactual DB). Added convenience symlinks: /var/www/{webroot, config}. Set MySQL root password on firstboot (convenience, security). Force MySQL to use Unicode/UTF8. Added php-xcache PHP opcode cacher / optimizer (performance). Set postfix MTA myhostname to localhost (bugfix). Enabled tracking in PHPMyAdmin. Major component versions projectpier 0.8.0.3 (upstream zipfile) mysql-server 5.1.41-3ubuntu12.8 apache2 2.2.14-5ubuntu8.4 phpmyadmin 4:3.3.2-1 Note: Please refer to turnkey-core's changelog for changes common to all appliances. Here we only describe changes specific to this appliance. Links Release meta-files (signature, manifest) [Less]

Appliance:  PrestaShop Appliance Changes: Initial public release of TurnKey Linux PrestaShop. PrestaShop related: Set admin email and password on firstboot (convenience ... [More] , security). Administration link set to: https://$your_domain/administration Optimized SEO URL's enabled by default. MySQL related: Set MySQL root password on firstboot (convenience, security). Force MySQL to use Unicode/UTF8. Includes PhpMyAdmin (listening on port 12322 - uses SSL). SSL support out of the box. Regenerates all secrets during installation / firstboot (security). Includes php-xcache PHP opcode cacher / optimizer (performance). Includes postfix MTA (bound to localhost) for sending of email (e.g. password recovery). Also includes webmin postfix module for convenience. Major component versions prestashop 1.3.3.0 (upstream tarball) mysql-server 5.1.41-3ubuntu12.8 apache2 2.2.14-5ubuntu8.4 phpmyadmin 4:3.3.2-1 Note: Please refer to turnkey-core's changelog for changes common to all appliances. Here we only describe changes specific to this appliance. Links Release meta-files (signature, manifest) [Less]

Appliance:  Revision Control Appliance Changes: Transitioned to upstream version of loggerhead. Still using custom loggerhead init script (flexibility, stability). Major ... [More] component versions git-core 1:1.7.0.4-1ubuntu0.1 gitweb 1:1.7.0.4-1ubuntu0.1 bzr 2.1.1-1 loggerhead 1.17+bzr400-1 subversion 1.6.6dfsg-2ubuntu1 websvn 2.3.0-1 mercurial 1.4.3-1 apache2 2.2.14-5ubuntu8.4 build-essential 11.4build1 Note: Please refer to turnkey-core's changelog for changes common to all appliances. Here we only describe changes specific to this appliance. Links Release meta-files (signature, manifest) [Less]

Appliance:  SDK for Amazon EC2 Changes: Installed python-boto, ruby-rightscale from upstream packages. Included AWS EC2 perl library. Mirrored latest python-boto documentation ... [More] for offline viewing. Mirrored latest AWS documentation for offline viewing. Tweaked TKL web control panel. Major component versions python-boto 1.9b-1ubuntu3 libright-aws-ruby 1.10.0-0ubuntu1 libnet-amazon-ec2-perl 0.13-1 ec2-ami-tools 1.3-45758-0ubuntu1 ec2-api-tools 1.3.46266-0ubuntu1 ruby-full 4.2 ipython 0.10-1ubuntu1 build-essential 11.4build1 openjdk-6-jre-headless 6b20-1.9.2-0ubuntu1~10.04.1 ntpdate 1:4.2.4p8+dfsg-1ubuntu2 lighttpd 1.4.26-1.1ubuntu3 Note: Please refer to turnkey-core's changelog for changes common to all appliances. Here we only describe changes specific to this appliance. Links Release meta-files (signature, manifest) [Less]

Appliance:  Torrent Server Appliance Changes: Set passwords on firstboot (convenience, security). Major component versions mldonkey-server 3.0.1-1 ... [More] webgmui 0.1.3 (upstream tarball) clamav-daemon 0.96.1+dfsg-0ubuntu0.10.04.2 stunnel4 3:4.29-1 samba 2:3.4.7~dfsg-1ubuntu3.2 extplorer 2.0.1 (upstream zipfile) vsftpd 2.2.2-3ubuntu6 lighttpd 1.4.26-1.1ubuntu3 Note: Please refer to turnkey-core's changelog for changes common to all appliances. Here we only describe changes specific to this appliance. Links Release meta-files (signature, manifest) [Less]

Appliance:  Symfony Appliance Changes: Set MySQL root password on firstboot (convenience, security). Force MySQL to use Unicode/UTF8. Added php-xcache PHP opcode cacher / ... [More] optimizer (performance). Set postfix MTA myhostname to localhost (bugfix). Enabled tracking in PHPMyAdmin. Major component versions symfony 1.4.8 (upstream tarball) php-pear 5.3.2-1ubuntu4.5 mysql-server 5.1.41-3ubuntu12.8 apache2 2.2.14-5ubuntu8.4 phpmyadmin 4:3.3.2-1 Note: Please refer to turnkey-core's changelog for changes common to all appliances. Here we only describe changes specific to this appliance. Links Release meta-files (signature, manifest) [Less]

Appliance:  Tracks Appliance Changes: Replaced Ruby with Ruby Enterprise (performance). Set MySQL root password on firstboot (convenience, security). Force MySQL to use ... [More] Unicode/UTF8. Set postfix MTA myhostname to localhost (bugfix). Included libreadline-dev so rails console works (bugfix). Major component versions tracks 1.7.2 (upstream tarball) ruby-enterprise 1.8.7-2010.02_i386_ubuntu10.04 mysql-server 5.1.41-3ubuntu12.8 apache2 2.2.14-5ubuntu8.4 build-essential 11.4build1 Note: Please refer to turnkey-core's changelog for changes common to all appliances. Here we only describe changes specific to this appliance. Links Release meta-files (signature, manifest) [Less]

Every now and then Liraz and I find ourselves chatting about how much we love Python, but more so the lessons we have learned from coding, and how to apply them to create beautiful Python code. I've tried to refine some of the "lessons" into ... [More] practical guidelines that I apply religiously to all new code I write, and the refactoring of old code I written. When reading other peoples code it sometimes ties my mind into knots, and on occasions I want to pull my hair out in frustration and disgust.  That's not to say I'm perfect, but hopefully these guidelines will benefit others (and indirectly help reduce my hair loss). I couldn't possibly include everything I wanted to in one post, so this will be the first, and more will follow...   #1 - OO structure == Well defined mental concepts Object Orientated structure should always map to well defined mental concepts in the problem domain. If you don't have a well defined mental model of the problem domain, start with that. Class Responsibility Collaboration (CRC) cards are really useful in this.   Basically you need a sketch of the architecture. What parts does your system have, what are their names, what does each part do? What parts is that part made out of? How do the parts interact with each other?   You can save quite a bit of effort if you come up with a good architecture up-front, but sometimes it may be easier to start without and figure it out a bit later after you have a bit more knowledge about the problem you are solving.   The rule is that the sooner you refine your architecture, the better. It is easy to dig yourself further and further into a complexity hole that makes restructuring very difficult later on. So do that as early as possible.   Refining the architecture is part of the "refactor mercilessly" rule.   #2 - Leverage built-in Python types It is often a good idea to build on top of built-in Python types or at least emulate them.   The big advantage in building on top of Python's conventions is that you get to re-use Python's abstractions instead of reinventing your own.  Python is famous for its minimal elegance, and the language designers take great care making small beautiful data structures. Making your code structures more Pythonic is a pretty good way to leverage the built-in elegance of the language while saving yourself quite a bit of work (inventing good abstractions is hard!).   If you've never inherited from a built-in data type, experiment with a small test case in a throw away script. This way you don't mangle your current project and you can be as experimental as you like.   Tip: Construction can be a bit trickier than a normal object if your initialization interface is different from the built-in type. In that case you'll need to override the __new__ magic method as well to call the __new__ constructor of the base class with different arguments than your __new__ is called.   For example: class IntField(int): def __new__(cls, val, name): return int.__new__(cls, val) def __init__(self, val, name): self.name = name self.val = val int.__init__(self, val)   As for emulating built in types, Python provides magical method names for emulating any built-in behavior. See special method names for reference.   #3 - Use the class namespace, Luke! Use the class namespace when defining class-level attributes and the instance namespace (which inherits from the class namespace on initialization) when defining instance level attributes.   For example, constants are always class level attributes because they are shared by all instances. On a practical level there are two reasons to do this: Enable inheritance - you can't override instance level attributes, only class level attributes. Readability - code is communication. Setting an attribute at the class level or instance level is making a statement about the nature of that attribute, which makes the code easier to read and understand.   #4 - staticmethod vs. classmethod vs. regular method Static methods are simpler and more readable than class methods because they don't have access to the class attribute, so it's easier to determine its input (i.e., arguments), and output (i.e., the return value).   Class methods are simpler than regular methods because the convention is that you don't usually manipulate attributes in the class namespace the way you would manipulate attributes in the instance namespace. So when you call a class method you know its not going to be sneaking around behind you back reading or writing to any instance level attributes set by other methods. The input for a class method is therefore the arguments it receives + class level constants. Its output is the return value.   A regular method is the most complex and easy to abuse type of method.  Its harder for the programmer to see what the inputs for the method are, and what its output is.   Lets consider the following case: imagine a class in which all private methods are called without arguments and do not return output values.  Instead, the private methods freely read and write to any attributes in self.   The problem with such a pattern, is that all methods have now become entangled in a spaghetti like dependency structure. The instance namespace in effect behaves like a global namespace and it becomes very difficult to understand methods in terms of input and output.  Furthermore, the boundaries between methods can easily become fuzzy and unclear.   You must never ever do this (I have in the passed, and I'm ashamed). If your code exhibits this pattern, go fix it, now!   Private methods have input arguments and return values for a reason. The instance namespace must only be used for instance level attributes which need to persist after a public method has been called.   Until it becomes second nature, these rules should help: If a method doesn't need access to instance level attributes then it should be a class method, not a regular method. If a method doesn't need access to class level attributes then it should be a static method, not a class method.   The Python Paradox Paul Graham wrote an interesting essay entitled The Python Paradox. It's a quick read - take a look.   [Less]

I'd like to introduce Joe. He is a good looking, experienced sys-admin and like all good sysadmins, he has more stuff to do than time to do it. Joe wants to get up and running on Amazon EC2 with a Wordpress installation, and chooses to do so ... [More] with a pre-configured appliance. These are the steps Joe performs: Joe logs into his favourite Amazon EC2 console, specifies a Wordpress appliance, and other configurations. Clicks launch. Once the instance is running, he logs in using his SSH public key and changes the root password (it was set randomly on firstboot, right?). He then proceeds to change the MySQL root password as well (also set randomly on firstboot, hopefully!). Joe knows how to do this as he's an experienced sys-admin, do you? Finally, Joe logs into Wordpress using the default admin password (he noted the default password in the release notes before launching), resets the password and specifies his own email for the account. While performing the above, Joe was holding his breath and working as fast as he could because he was previously hit by a botnet looking for random systems using default passwords and was compromised. Luckily this time he came out unscaved. Does this sound familiar? Well, it should because that's how it's mostly being done. You might be thinking to yourself "but I used the TurnKey Hub to set the root password for my instances, which also set the database password". True, that has been a feature of the Hub from day one, but with the release of TurnKey 11.0 and the end to default passwords, we've extended the Hub to support preseeding as well. The idea behind this was not only to make cloud deployments more secure, but to make it much easier. We wanted to simplify the process for Joe from the above to this: Joe logs into the Hub, selects Wordpress and preseeds the configuration. Clicks launch. The above is not a mock-up of a future implementation, it's live on the Hub. So how does it work? Read on...   Brainstorming a solution The problem in preseeding an instance is sending the information (securely) to the instance. So how do you do it?   Idea #1: pass it through Amazon EC2 user-data? If you know a little about Amazon EC2 you'll know that when launching an instance you can specify user-data which is accessible from the instance via Amazon's API.   But wait, do you really want to store authentication credentials in user-data?    You could, but because any process on the instance that can open a network socket can access the user-data as it never expires, you'll probably want to firewall off the Amazon API as soon as it's not required anymore during instance initialization. But maybe the user of the instance needs access to the Amazon API? Crippling the service by design isn't a good solution in my honest opinion.   Idea #2: store it in the Hub's database, and let the server query the API So, instead of sending authentication credentials via user-data, why not send a unique identifier (e.g., SERVER_ID), so the instance can use the Hub's API to pull the credentials?   Well, you could, but that would mean the Hub service needs to store the instance's configuration, passwords and all, in its database and delete it when it's no longer needed. Storing an item in a database for just one use is inelegant. But it's a natural solution if you only have a database, as I dicussed in a previous blog post, "when all you have is a hammer, everything looks like a nail".   In my opinion, it ultimately comes down to separation of concerns. For this type of pattern, the most natural solution would be some sort of messaging service. The Hub publishes a message to a queue, which the instance consumes.   Idea #3: pass it as messages using the Advanced Message Queuing Protocol (AMQP) So whats wrong with messaging? Nothing really, so long as you take care when designing the system for confidentiality and integrity - we don't want others eavesdropping on messages, or sending spoofed messages.   Messages that fail a CRC or cannot be decrypted successfully should be discarded, and removed from the queue so not to block it.   Designing infrastructure that is secure, scalable and extendible The solution we came up with is designed to be secure, scalable and extendible. Eventually it will support other cloud hosting providers, as well as provide bi-directional secure communication for future Hub-based services still under development.   The solution uses each of the brainstormed solutions above for what they were designed for, and no more.   Data Flow Diagram (DFD) explained: The user specifies preseeding data. The Hub tells Amazon EC2 to launch the instance with user-data which includes the SERVER_ID. The Hub creates a direct message exchange and queue for the server, which is configured to only receive messages sent from the Hub. The Hub publishes symmetrically encrypted messages (incl. a CRC) to the server queue with preseeding data that only the server can decrypt. The instance pulls user-data from the Amazon EC2 API (SERVER_ID). The Instance registers itself with the Hub via an SSL secured API using the SERVER_ID, which responds back with the server subkey and messaging secret. Note that this can only be done once for security. subkey: A one way hash generated from the user's APIKEY. It is unique for each server registered in the Hub, and is used as part of the exchange and queue naming structure. secret: A secure unique hash used for message encryption and decryption. The instance consumes messages from the queue. Messages are decrypted and passed to the callback for processing (preseeding messages appends the arg=value to inithooks.conf). During inithooks execution, inithooks.conf is sourced so preseeding will happen. Once inithooks.conf is no longer needed, it is deleted. In addition to authentication related preseeding, TKLBAM is also preseeded with the HUB_APIKEY and is initialized, so performing the initial backup is as easy as executing tklbam-backup or using the TKLBAM webmin module. As always, the client side code that implements the above is open source, and can be found in the following projects: hubclient, tklamq, inithooks, as well as the above mentioned blog posts. Take the Hub for a spin and let us know what you think. [Less]