cloudify

Closed Loop Feedback Test Cloudify is an open source PaaS software stack, that automates deployment, monitoring and fault detection of applications running on the cloud. It can automatically add instances when monitored statistics exceeds a certain ... [More] threshold. This automatic scaling rules algorithm implementation is a closed-loop control system which requires careful testing. The diagram below shows the test load generator, and the closed loop “system under test”. Each web server’s throughput is being monitored by the controller (scaling rules). When throughput exceeds a certain threshold a new web server instance is started, and the throughput per instance goes down below threshold. +--------------+ | Test | | + | | +------+-------+ | http | requests V +--------------+ +--------------+ | Tomcat | | Throughput | | Instance(s) +--->| JMX | | | | Monitor | +--------------+ +-------+------+ ^ | | add | | instance | +------+-----+ | | Scaling | | | Rule |<-------------+ | | +------------+ Here is the JMX plugin configured to expose the Total Number of web Requests (per instance) https://gist.github.com/2788838 The scaling rule uses a 20 seconds sliding window to convert the Total number of Requests (X) into Throughput (delta X divided by delta T) and compares the result against a predefined threshold. https://gist.github.com/2788840 The closed loop test starts with “zero traffic” changes to “constant traffic” and then back to “zero traffic”: Start without any web traffic Wait until minimum number of instances. Increase web traffic to a predefined level of requests per second Wait until expected number of instances. Wait a little more, make sure no add/remove instance fluctuations. Stop all web traffic. Wait until minimum number of instances. The test waits until the expected number of instances is reached (step 4), and stays there for certain period of time (step 5). During that time we must verify that the scale out is performed without fluctuations. An unwanted fluctuation is when without any input change (stable input http traffic) an instance is added and then removed by the controller. In more advanced test scenarios we may want to monitor resources such as number of busy threads, or CPU usage. This would require a more sophisticated HTTP load generator, which is usually used in stress/performance testing. Open Loop Test The problem with developing a closed loop feedback system is that you cannot test the controller (scaling rules) in an isolated environment. Every decision the controller makes affects the output of the system which affects the controller. The way to deal with that is to “open” the  loop (non-feedback controller). The controller takes a decision, but the result does not affect the monitored data feeding the controller. +--------------+ | Test | | + | | +------+-------+ | set | value V +--------------+ +--------------+ | Stub | | value | | Instance(s) | | monitor | | | | | +--------------+ +------+-------+ ^ | | add | | instance | +------+-----+ | | Scaling | | | Rule |<------------+ | | +------------+ Here is a little Cloudify recipe trick. Each Cloudify instance stores the recipe as a POJO in memory, which allows adding new properties. In this case we add a long value which mocks the web server throughput. https://gist.github.com/2788844 This recipe allows the test to remotely inject the monitored values that the service exposes to the scaling rules controller. This mock value is not affected by the scaling rules decisions and does not require any actual web server instance running. Here is how an open loop controller test looks like: Set monitored value to 0 Wait for minimum number of instances. Set monitored value to “$highthreshold+1” Wait for maximum number of instances. Set monitored value to 0 Wait for minimum number of instances. Notice that step 4 expects the scaling out to be performed again and again until the maximum number of instances is reached. This is since there is no closed loop feedback. No matter how many web instances the scaling rules start, there would always see the same monitored value (greater than high threshold). [Less]

PaaS, Does It Really Mean No-Ops? I’d like to start with a brief overview of the evolution of the cloud… from Cloudify Community http://bit.ly/LvG5m1

I’m very excited to announce our first OpenStack Israel event on Wednesday the 30th of May in Petach Tikva in collaboration with the IGT Cloud and Rackspace. Since Avner Algom and myself started to work on the event a few… via the GigaSpaces Blog http://bit.ly/JTxbyv

Cloudify can start and stop machines automatically based on real-time statistics. The first step is to define per instance monitoring metrics. This metric defined in the recipe could be based on a JMX plugin, HTTP request, custom commandline output ... [More] , or custom groovy code. Statistics are used to normalize data and provide a figure we can compare against the threshold. The metrics are aggregated over time (per instance time average in the example below). The per-instance statistics are then aggregated again (maximum in the example below). This results with a service statistics that represents the whole cluster (maximum of averages). Which finally leads us to the scaling rules. When statistics is below the low threshold a scale-in operation is triggered (remove an existing instance). When statistics is above the high threshold a scale-out operation is triggered (adds a new instance). The scaling rules are bounded by the minimum total number of service instances, and maximum number of instances. Certain precaution needs to be taken after an instance has been started. It may take some time until the new instance metrics are usable. The instance could be getting low traffic due to load-balancer session stickiness, or due to cache warm-up. During that time it would be wrong to trigger another scaling rule. The cooldown period disables the scaling rules until the instance has been started and warmed up. For more information, consult the scaling rules documentation: http://www.cloudifysource.org/guide/developing/scaling_rules. [Less]

When integrating with cloud providers that expose REST API, and invoking it from my Cloud Driver Java-based implementation, I found the Apache Jersey client open source library quite convenient for implementing a RESTful client. Jersey is based ... [More] on JAX-RS Java community standard, and offers easy handling of various flavors of calls, cookie handling, policy governance, etc. Here is an example for invoking a POST call, using a JSON to relay invocation parameters, and using a token-based authentication: https://gist.github.com/2717455 For the full source code: OpenstackCloudDriver.java Here is an example of invoking a GET call, using explicit parameter listing and a session-cookie-based authentication: https://gist.github.com/2717520 The parsing of the XML response is done using DOM. [Less]

Cloudify uses recipes to describe service deployment. The syntax is very minimal, but allows writing custom groovy extensions. The recipe below shows how two tomcats running on the localcloud (your laptop) perform port resolution, and port detection. ... [More] Notice how context (isLocalCloud) is available, configuration properties are injected (httpPort) and ServiceUtils is imported automatically : https://gist.github.com/2694919 The parsing of the recipe is done with groovy, but results with a  Plain Old Java Object: https://gist.github.com/2694906 So how is this all possible? The parser uses the GroovyShell object to read the recipe text file. The gist below shows how to configure the classpath, custom properties, default imports, and a context object: https://gist.github.com/2694894 The gists in this posts have been syndicated from the Cloudify source code. This  code, written by Barak, is opensource ofcourse. The service recipe (DSL) and resulting POJO: tomcat-service.groovy Service.javaThe service parser and base class doing the magic: DSLReader.java BaseDSLScript.java [Less]

“Smilde says that his installations, which were developed for the online gallery probe, give form to “physical presence found within transitional space.” In this way, the works exist more fully as ideas committed to photographs, living evidence of the thaumaturgic power of the artist.” - http://www.architizer.com/en_us/blog/dyn/39784/39784/

Ops: For the sake of this discussion... What do you mean when you say "private cloud, public cloud and data center"? Cloudify: A data center is a bunch of machines that are running most of the time. It could be a bare-bones data center or a ... [More] virtualized data-center. A private cloud is a data center that has an API for provisioning machines on-demand. It takes away the need to manually decide which VM runs on which physical machine for which user. A public cloud is like a private cloud, but each machine is billed by the hour. Ops: Ok, so how do you provision service instances to machines? Cloudify: The simplest scenario for clouds is to start a new virtual machine before an instance starts, and stop the virtual machine after an instance stops. Ops: And what if I don't have a cloud? Cloudify: In that case the machines are always running, and the instance is started on a machine that is not used by any other instance. When the instance is stopped the machine becomes vacant (and another instance can use it). Ops: On a public cloud I am billed for each machine by the hour. If I use a machine for only 20 minutes and start a new machine for 20 minutes, I pay double the price. Cloudify vFuture: In that case, we use a mixed strategy. Before a new instance is started, look for a vacant machine. If there is no vacant machine start one. When the instance is stopped the machine becomes vacant (and another instance can use it). The machine is left vacant until its hourly billing period is over and then it's stopped. Ops: So while it is vacant, if I start another instance then it will not start a new machine, but rather use the existing machine. Cloudify vFuture: Correct. This time sharing mode saves money. The downside of this is that when the instance is stopped, the machine is not deleted and another instance could sniff it later. Ops: Got it. So if I don't start another instance the machine is left vacant until the end of the billing hour? Cloudify vFuture: Basically yes. You can define for each service the must-have number of instances, and nice-to-have number of instances. This time if a vacant machine is scheduled to go down in 20 minutes, (and there is no must-have instance) a nice-to-have instance will use the machine for the remaining 20 minutes. Ops: And what about multi-tenant scenarios? Cloudify vFuture: Each service instance runs in a separate process (instead of a separate machine). An instance can share the same machine with other processes as long as they serve the same tenant. In a loose security environment you can allow different processes that serve different tenants to run on same machines. [Less]

We run our Machines Service Level Agreement component test against XenServer. Please tweet @itaifrenkel if you can translate this log file from Estonian.

Automated testing of Cloudify for Azure