CUBRID Database Management System

When there is an obstacle, or when a Java based Web application is running much slower than expected, we need to use thread dumps. If thread dumps feel like very complicated to you, this article may help you very much. Here I will explain what ... [More] threads are in Java, their types, how they are created, how to manage them, how you can dump threads from a running application, and finally how you can analyze them and determine the bottleneck or blocking threads. This article is a result of long experience in Java application debugging. Java and ThreadA web server uses tens to hundreds of threads to process a large number of concurrent users. If two or more threads utilize the same resources, a contention between the threads is inevitable, and sometimes deadlock occurs. Thread contention is a status in which one thread is waiting for a lock, held by another thread, to be lifted. Different threads frequently access shared resources on a web application. For example, to record a log, the thread trying to record the log must obtain a lock and access the shared resources. Deadlock is a special type of thread contention, in which two or more threads are waiting for the other threads to complete their tasks in order to complete their own tasks. Different issues can arise from thread contention. To analyze such issues, you need to use the thread dump. A thread dump will give you the information on the exact status of each thread. Background Information for Java ThreadsThread SynchronizationA thread can be processed with other threads at the same time. In order to ensure compatibility when multiple threads are trying to use shared resources, one thread at a time should be allowed to access the shared resources by using thread synchronization. Thread synchronization on Java can be done using monitor. Every Java object has a single monitor. The monitor can be owned by only one thread. For a thread to own a monitor that is owned by a different thread, it needs to wait in the wait queue until the other thread releases its monitor. Thread StatusIn order to analyze a thread dump, you need to know the status of threads. The statuses of threads are stated on java.lang.Thread.State.   Figure 1: Thread Status. NEW: The thread is created but has not been processed yet. RUNNABLE: The thread is occupying the CPU and processing a task. (It may be in WAITING status due to the OS's resource distribution.) BLOCKED: The thread is waiting for a different thread to release its lock in order to get the monitor lock. WAITING: The thread is waiting by using a wait, join or park method. TIMED_WAITING: The thread is waiting by using a sleep, wait, join or park method. (The difference from WAITING is that the maximum waiting time is specified by the method parameter, and WAITING can be relieved by time as well as external changes.)  Thread TypesJava threads can be divided into two: daemon threads; and non-daemon threads. Daemon threads stop working when there are no other non-daemon threads. Even if you do not create any threads, the Java application will create several threads by default. Most of them are daemon threads, mainly for processing tasks such as garbage collection or JMX. A thread running the 'static void main(String[] args)’ method is created as a non-daemon thread, and when this thread stops working, all other daemon threads will stop as well. (The thread running this main method is called the VM thread in HotSpot VM.) Getting a Thread DumpWe will introduce the three most commonly used methods. Note that there are many other ways to get a thread dump. A thread dump can only show the thread status at the time of measurement, so in order to see the change in thread status, it is recommended to extract them from 5 to 10 times with 5-second intervals. Getting a Thread Dump Using jstackIn JDK 1.6 and higher, it is possible to get a thread dump on MS Windows using jstack. Use PID via jps to check the PID of the currently running Java application process. [user@linux ~]$ jps -v25780 RemoteTestRunner -Dfile.encoding=UTF-825590 sub.rmi.registry.RegistryImpl 2999 -Dapplication.home=/home1/user/java/jdk.1.6.0_24 -Xms8m26300 sun.tools.jps.Jps -mlvV -Dapplication.home=/home1/user/java/jdk.1.6.0_24 -Xms8m Use the extracted PID as the parameter of jstack to obtain a thread dump. [user@linux ~]$ jstack -f 5824  A Thread Dump Using jVisualVMGenerate a thread dump by using a program such as jVisualVM. Figure 2:  A Thread Dump Using visualvm. The task on the left indicates the list of currently running processes. Click on the process for which you want the information, and select the thread tab to check the thread information in real time. Click the Thread Dump button on the top right corner to get the thread dump file. Generating in a Linux TerminalObtain the process pid by using ps -ef command to check the pid of the currently running Java process. [user@linux ~]$ ps - ef | grep javauser      2477          1    0 Dec23 ?         00:10:45 ...user    25780 25361   0 15:02 pts/3    00:00:02 ./jstatd -J -Djava.security.policy=jstatd.all.policy -p 2999user    26335 25361   0 15:49 pts/3    00:00:00 grep java Use the extracted pid as the parameter of kill –SIGQUIT(3) to obtain a thread dump. Thread Information from the Thread Dump File "pool-1-thread-13" prio=6 tid=0x000000000729a000 nid=0x2fb4 runnable [0x0000000007f0f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) - locked <0x0000000780b7e688> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:167) at java.io.BufferedReader.fill(BufferedReader.java:136) at java.io.BufferedReader.readLine(BufferedReader.java:299) - locked <0x0000000780b7e688> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(BufferedReader.java:362) Thread name: When using Java.lang.Thread class to generate a thread, the thread will be named Thread-(Number), whereas when using java.util.concurrent.ThreadFactory class, it will be named pool-(number)-thread-(number). Priority: Represents the priority of the threads. Thread ID: Represents the unique ID for the threads. (Some useful information, including the CPU usage or memory usage of the thread, can be obtained by using thread ID.) Thread status: Represents the status of the threads. Thread callstack: Represents the call stack information of the threads.  Thread Dump Patterns by Type When Unable to Obtain a Lock (BLOCKED) This is when the overall performance of the application slows down because a thread is occupying the lock and prevents other threads from obtaining it. In the following example, BLOCKED_TEST pool-1-thread-1 thread is running with <0x0000000780a000b0> lock, while BLOCKED_TEST pool-1-thread-2 and BLOCKED_TEST pool-1-thread-3 threads are waiting to obtain <0x0000000780a000b0> lock. Figure 3: A thread blocking other threads.  "BLOCKED_TEST pool-1-thread-1" prio=6 tid=0x0000000006904800 nid=0x28f4 runnable [0x000000000785f000] java.lang.Thread.State: RUNNABLE at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:282) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) - locked <0x0000000780a31778> (a java.io.BufferedOutputStream) at java.io.PrintStream.write(PrintStream.java:432) - locked <0x0000000780a04118> (a java.io.PrintStream) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272) at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85) - locked <0x0000000780a040c0> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168) at java.io.PrintStream.newLine(PrintStream.java:496) - locked <0x0000000780a04118> (a java.io.PrintStream) at java.io.PrintStream.println(PrintStream.java:687) - locked <0x0000000780a04118> (a java.io.PrintStream) at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:44) - locked <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState) at com.nbp.theplatform.threaddump.ThreadBlockedState$1.run(ThreadBlockedState.java:7) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000780a31758> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "BLOCKED_TEST pool-1-thread-2" prio=6 tid=0x0000000007673800 nid=0x260c waiting for monitor entry [0x0000000008abf000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:43) - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState) at com.nbp.theplatform.threaddump.ThreadBlockedState$2.run(ThreadBlockedState.java:26) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000780b0c6a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "BLOCKED_TEST pool-1-thread-3" prio=6 tid=0x00000000074f5800 nid=0x1994 waiting for monitor entry [0x0000000008bbf000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:42) - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState) at com.nbp.theplatform.threaddump.ThreadBlockedState$3.run(ThreadBlockedState.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000780b0e1b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) When in Deadlock Status This is when thread A needs to obtain thread B's lock to continue its task, while thread B needs to obtain thread A's lock to continue its task. In the thread dump, you can see that DEADLOCK_TEST-1 thread has 0x00000007d58f5e48 lock, and is trying to obtain 0x00000007d58f5e60 lock. You can also see that DEADLOCK_TEST-2 thread has 0x00000007d58f5e60 lock, and is trying to obtain 0x00000007d58f5e78 lock. Also, DEADLOCK_TEST-3 thread has 0x00000007d58f5e78 lock, and is trying to obtain 0x00000007d58f5e48 lock. As you can see, each thread is waiting to obtain another thread's lock, and this status will not change until one thread discards its lock. Figure 4: Threads in a Deadlock status. "DEADLOCK_TEST-1" daemon prio=6 tid=0x000000000690f800 nid=0x1820 waiting for monitor entry [0x000000000805f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197) - waiting to lock <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182) - locked <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers: - None "DEADLOCK_TEST-2" daemon prio=6 tid=0x0000000006858800 nid=0x17b8 waiting for monitor entry [0x000000000815f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197) - waiting to lock <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182) - locked <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers: - None "DEADLOCK_TEST-3" daemon prio=6 tid=0x0000000006859000 nid=0x25dc waiting for monitor entry [0x000000000825f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197) - waiting to lock <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182) - locked <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers: - None When Continuously Waiting to Receive Messages from a Remote Server The thread appears to be normal, since its state keeps showing as RUNNABLE. However, when you align the thread dumps chronologically, you can see that socketReadThread thread is waiting infinitely to read the socket. Figure 5: Continuous Waiting Status. "socketReadThread" prio=6 tid=0x0000000006a0d800 nid=0x1b40 runnable [0x00000000089ef000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) - locked <0x00000007d78a2230> (a java.io.InputStreamReader) at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:107) - locked <0x00000007d78a2230> (a java.io.InputStreamReader) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:93) at java.io.InputStreamReader.read(InputStreamReader.java:151) at com.nbp.theplatform.threaddump.ThreadSocketReadState$1.run(ThreadSocketReadState.java:27) at java.lang.Thread.run(Thread.java:662) When Waiting The thread is maintaining WAIT status. In the thread dump, IoWaitThread thread keeps waiting to receive a message from LinkedBlockingQueue. If there continues to be no message for LinkedBlockingQueue, then the thread status will not change. Figure 6: Waiting status. "IoWaitThread" prio=6 tid=0x0000000007334800 nid=0x2b3c waiting on condition [0x000000000893f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007d5c45850> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:440) at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:629) at com.nbp.theplatform.threaddump.ThreadIoWaitState$IoWaitHandler2.run(ThreadIoWaitState.java:89) at java.lang.Thread.run(Thread.java:662) When Thread Resources Cannot be Organized NormallyUnnecessary threads will pile up when thread resources cannot be organized normally. If this occurs, it is recommended to monitor the thread organization process or check the conditions for thread termination. Figure 7: Unorganized Threads. How to Solve Problems by Using Thread DumpExample 1: When the CPU Usage is Abnormally High Extract the thread that has the highest CPU usage.[user@linux ~]$ ps -mo pid.lwp.stime.time.cpu -C java     PID         LWP          STIME                  TIME        %CPU10029               -         Dec07          00:02:02           99.5         -       10039        Dec07          00:00:00              0.1         -       10040        Dec07          00:00:00           95.5 From the application, find out which thread is using the CPU the most. Acquire the Light Weight Process (LWP) that uses the CPU the most and convert its unique number (10039) into a hexadecimal number (0x2737). After acquiring the thread dump, check the thread's action. Extract the thread dump of an application with a PID of 10029, then find the thread with an nid of 0x2737. "NioProcessor-2" prio=10 tid=0x0a8d2800 nid=0x2737 runnable [0x49aa5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x74c52678> (a sun.nio.ch.Util$1) - locked <0x74c52668> (a java.util.Collections$UnmodifiableSet) - locked <0x74c501b0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at external.org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65) at external.org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:708) at external.org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Extract thread dumps several times every hour, and check the status change of the threads to determine the problem. Example 2: When the Processing Performance is Abnormally Slow   After acquiring thread dumps several times, find the list of threads with BLOCKED status. " DB-Processor-13" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f000] java.lang.Thread.State: BLOCKED (on object monitor) at beans.ConnectionPool.getConnection(ConnectionPool.java:102) - waiting to lock <0xe0375410> (a beans.ConnectionPool) at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111) at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43) "DB-Processor-14" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f020] java.lang.Thread.State: BLOCKED (on object monitor) at beans.ConnectionPool.getConnection(ConnectionPool.java:102) - waiting to lock <0xe0375410> (a beans.ConnectionPool) at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111) at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43) " DB-Processor-3" daemon prio=5 tid=0x00928248 nid=0x8b waiting for monitor entry [0x000000000825d080] java.lang.Thread.State: RUNNABLE at oracle.jdbc.driver.OracleConnection.isClosed(OracleConnection.java:570) - waiting to lock <0xe03ba2e0> (a oracle.jdbc.driver.OracleConnection) at beans.ConnectionPool.getConnection(ConnectionPool.java:112) - locked <0xe0386580> (a java.util.Vector) - locked <0xe0375410> (a beans.ConnectionPool) at beans.cus.Cue_1700c.GetNationList(Cue_1700c.java:66) at org.apache.jsp.cue_1700c_jsp._jspService(cue_1700c_jsp.java:120) Acquire the list of threads with BLOCKED status after getting the thread dumps several times. If the threads are BLOCKED, extract the threads related to the lock that the threads are trying to obtain. Through the thread dump, you can confirm that the thread status stays BLOCKED because <0xe0375410> lock could not be obtained. This problem can be solved by analyzing stack trace from the thread currently holding the lock. There are two reasons why the above pattern frequently appears in applications using DBMS. The first reason is inadequate configurations. Despite the fact that the threads are still working, they cannot show their best performance because the configurations for DBCP and the like are not adequate. If you extract thread dumps multiple times and compare them, you will often see that some of the threads that were BLOCKED previously are in a different state. The second reason is the abnormal connection. When the connection with DBMS stays abnormal, the threads wait until the time is out. In this case, even after extracting the thread dumps several times and comparing them, you will see that the threads related to DBMS are still in a BLOCKED state. By adequately changing the values, such as the timeout value, you can shorten the time in which the problem occurs. Coding for Easy Thread DumpNaming ThreadsWhen a thread is created using java.lang.Thread object, the thread will be named Thread-(Number). When a thread is created using java.util.concurrent.DefaultThreadFactory object, the thread will be named pool-(Number)-thread-(Number). When analyzing tens to thousands of threads for an application, if all the threads still have their default names, analyzing them becomes very difficult, because it is difficult to distinguish the threads to be analyzed. Therefore, you are recommended to develop the habit of naming the threads whenever a new thread is created.  When you create a thread using java.lang.Thread, you can give the thread a custom name by using the creator parameter. public Thread(Runnable target, String name);public Thread(ThreadGroup group, String name);public Thread(ThreadGroup group, Runnable target, String name);public Thread(ThreadGroup group, Runnable target, String name, long stackSize); When you create a thread using java.util.concurrent.ThreadFactory, you can name it by generating your own ThreadFactory. If you do not need special functionalities, then you can use MyThreadFactory as described below: import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.ThreadFactory;import java.util.concurrent.atomic.AtomicInteger;public class MyThreadFactory implements ThreadFactory {  private static final ConcurrentHashMap POOL_NUMBER =                                                       new ConcurrentHashMap();  private final ThreadGroup group;  private final AtomicInteger threadNumber = new AtomicInteger(1);  private final String namePrefix;   public MyThreadFactory(String threadPoolName) {            if (threadPoolName == null) {          throw new NullPointerException("threadPoolName");      }             POOL_NUMBER.putIfAbsent(threadPoolName, new AtomicInteger());            SecurityManager securityManager = System.getSecurityManager();      group = (securityManager != null) ? securityManager.getThreadGroup() :                                                    Thread.currentThread().getThreadGroup();            AtomicInteger poolCount = POOL_NUMBER.get(threadPoolName);      if (poolCount == null) {            namePrefix = threadPoolName + " pool-00-thread-";      } else {            namePrefix = threadPoolName + " pool-" + poolCount.getAndIncrement() + "-thread-";      }  }   public Thread newThread(Runnable runnable) {      Thread thread = new Thread(group, runnable, namePrefix + threadNumber.getAndIncrement(), 0);      if (thread.isDaemon()) {            thread.setDaemon(false);      }      if (thread.getPriority() != Thread.NORM_PRIORITY) {            thread.setPriority(Thread.NORM_PRIORITY);      }      return thread;  }} Obtaining More Detailed Information by Using MBeanYou can obtain ThreadInfo objects using MBean. You can also obtain more information that would be difficult to acquire via thread dumps, by using ThreadInfo. ThreadMXBean mxBean = ManagementFactory.getThreadMXBean();long[] threadIds = mxBean.getAllThreadIds();ThreadInfo[] threadInfos =                mxBean.getThreadInfo(threadIds);for (ThreadInfo threadInfo : threadInfos) {  System.out.println(      threadInfo.getThreadName());  System.out.println(      threadInfo.getBlockedCount());  System.out.println(      threadInfo.getBlockedTime());  System.out.println(      threadInfo.getWaitedCount());  System.out.println(      threadInfo.getWaitedTime());}  You can acquire the amount of time that the threads WAITed or were BLOCKED by using the method in ThreadInfo, and by using this you can also obtain the list of threads that have been inactive for an abnormally long period of time. In ConclusionIn this article I was concerned that for developers with a lot of experience in multi-thread programming, this material may be common knowledge, whereas for less experienced developers, I felt that I was skipping straight to thread dumps, without providing enough background information about the thread activities. This was because of my lack of knowledge, as I was not able to explain the thread activities in a clear yet concise manner. I sincerely hope that this article will prove helpful for many developers. By Tae Jin Gu, Software Engineer at Web Platform Development Lab, NHN Corporation. [Less]

NoSQL is the talk of the town. And we have already covered what it is for in one of our previous blogs. Today I would like to share the NoSQL benchmark test results we have recently conducted. It will help you to understand if the soon to develop ... [More] system is compatible to NoSQL, and which NoSQL product to select. In this article we will reveal the characteristics of Cassandra, HBase and MongoDB identified through multiple workloads. Why NoSQL? The interest in NoSQL continues to rise because the amount of data to process continues to increase. International internet companies, including Google and Facebook, have their own NoSQL solution designed to process the exploding amount of data. Why are they using NoSQL instead of RDBMS? Twitter is still using MySQL. Although smaller than Facebook, the international SNS company MySpace uses MS SQL Server as their main storage. RDBMS is known to experience burden when processing tera or peta unit large sized data. However, this can be resolved through sharding. For instance, NHN’s Café, Blog and News services use RDBMS through sharding. There is no single correct answer in processing bulk data. Since every situation is different, the company must select the solution that best fits their situation and apply that for seamless service. Out of the RDBMSs, Oracle is an exception since Oracle’s performance and functions, such as mass data processing or data synchronization, are far more superior to other RDBMS. However, the high cost can be a problem. Depending on the size, it may be more economical to develop a NoSQL than purchasing an Oracle license. Even if you have enough expertise on how to process bulk data with RDBMS, we will need to have continuous interest and training in NoSQL. NoSQL is gaining popularity because of its ability to process mass data, but it still has many technical (or functional) limitations compared to RDBMS. However, this will be resolved as time passes. NoSQL provides a non-relational, and in the long run, schema-free data model, allowing the horizontal extension of the system. Instead, it is less structured than RDBMS and does not guarantee ACID. Therefore, after the INSERT operation is completed and SELECT conducted, a different value can be acquired. And after the NoSQL demo has failed and restored, the stored value may be different than before. Operations such as transactions or join cannot be conducted. While ACID has been abandoned, the flexibility, scalability and usability of data storage have increased. Therefore, it is more suitable for explosive amounts of data processing. Now let’s see if the NoSQL products really play the expected part for the internet services you are to develop. Why don’t we take a look at the characteristics of the most widely used NoSQL products (Cassandra, HBase, MongoDB) by looking into each of its architectures and benchmark tests that were conducted. Benchmarking Tests using YCSB YCSB (Yahoo Cloud Servicing Benchmark) is a test framework developed by Yahoo. It allows to conduct benchmark tests on storages by creating "work loads". Through this benchmark, the storage most suitable for the service that is to be developed can be selected. Basic operations are Insert, Update, Read, and Scan. There are basic workload sets that combine the basic operations, but new additional workloads can also be created. YCBS currently supports Cassandra, HBase, MongoDB, Voldemort and JDBC. If tests on other storages are needed, then use YCBS interfaces to test during the development process. This article contains tests conducted on the following products and versions. Cassandra-0.7.4Although Cassandra’s latest version is 0.8.0, we have decided to use the previous version known to be stable. Because when testing with the 0.8 version, the gossip protocol between nodes malfunctioned and the node up/down information was incorrect. HBase-0.90.2 (Hadoop-0.20-append)The HBase-0.90.2 (Hadoop-0.20-append) was selected because, if not the Hadoop-append version, there may be problems on decreased durability in HDFS. MongoDB-1.8.1 The test workload is as follows. Insert OnlyEnter 50 million 1K-sized records to the empty DB. Read OnlySearch the key in the Zipfian Distribution for a one hour period on the DB that contains 50 million 1K-sized records. Read & UpdateConduct read and update one-on-one instead of read under the identical conditions of Read Only. There are three testing equipments with the same specifications: Nehalem 6 Core x 2 CPU, 16GB Memory. Each conduct replicates and distributes three copies. However, with MongoDB the performance result was abnormally high when both replications and distribution compositions were organized, so the test was conducted only with the replica set. The parameter for each product is shown in the following listings. Other items were set to default. Cassandra # If a minimum of one succeeds while replicating, then it is returned to application. Consistency Level, Read=ONE, Write=ONE # Can be set to periodic(default) or batch. # Periodically writes (fsync) the commit log on the disk, # and batch executes regularly (1ms) collecting those to fsync every time. commitlog_sync: batch, commitlog_sync_batch_window_in_ms: 1 # Degree the key location is cached. When 1.0 - everything is cached. key_cached=1.0 HBase HeapSize, HBase=8G, HDFS=2G MongoDB # Size of Oplog. Oplog is the log the master accumulated for replication. --oplogSize=100G The results are as shown in the following figure. All three products showed better throughput in INSERT. Cassandra showed outstanding throughput in INSERT-only with 20,000 ops. HBase also presented relatively good performance in INSERT-only. The products did not show large differences with READ-only than INSERT-only. HBase’s performance was better than Cassandra in READ-only. Cassandra’s performance in READ-and-UPDATE was better than the other two products. Cassandra’s READ-and-UPDATE might have been higher than READ-only due to Cassandra’s excellent insert throughout. MongoDB’s throughout in all three conditions was the lowest of the three products. MongoDB, which uses the Memory Mapped File (mmap), probably showed poor performance because the large data size exceeded the physical memory size. Let’s take a look at why each product showed such difference in performance. Cassandra and HBase Architecture Both Cassandra and HBase are influenced by Google’s BigTable. (Although Cassandra was directly influenced by Amazon’s Dynamo.) BigTable is a Column-Oriented DB that stores data in a Multidimensional Sorted Map format and has the data structure. Column family is a basic unit that stores data with column groups that are related to each other. In BigTable, data is written basically with the append method. In other words, when modifying data, the updates are appended to a file, rather than an in-place update in the stored file. The figure below shows BigTable’s data read/insert path. When a write operation is inserted, it is first placed in a memory space called memtable. If the memtable is full, then the whole data is stored in a file called SSTable (Sorted String Table). Data may be lost if the server collapses before the memtable data is moved to SSTable, so to provide durability it is necessary to save the history of commit logs every time before writing to the memtable. When conducting the read operation, first find the pertaining key in the memtable. If it is not in the memtable, search for it in the SSTable. You may have to search multiple SSTables. There are advantages in the writing operation if this architecture is used. This is because the ‘writing’ operation is only recorded in the memory and moved to the actual disk only after a certain amount has been accumulated. Therefore, concurrent I/O can be avoided. However, when "reading", if "reading" is done in the SSTable and not in the memtable, then the performance will relatively decrease. Both Cassandra and HBase use bloom filter to quickly judge whether the key exists in the SSTable and creates an index for use. However, if there are many SSTables, then a lot of I/O will be created during the reading operation. Therefore, compaction is done to improve the reading performance. Compaction is where two SSTables merge and sort to become one SSTable, which decreases the number of SSTables. The reading and writing performance improved as more compactions are done. For these reasons READ and READ-and-UPDATE are much slower than INSERT operations in these NoSQL solutions. Cassandra Additionally, Cassandra uses consistent hashing. The HBase and Casandra may have their similarities, there are differences as well. Cassandra prefers AP (Availability & Partition tolerance) while HBase prefers CP (Consistency & Partition tolerance). CAP theorem is a theory in Distributed Computing. The theory claims that there is no system that provides all three Consistency, Availability and Partition tolerance. For example, replications must be made in multiple nodes to increase usability, and the adjustability between replicated data must be met. However, in order to make operation possible, even during network partitioning, it becomes difficult to guarantee the adjustability between replications or data. Therefore, only a part of the CAP can be provided. As mentioned before, Amazon’s Dynamo has a direct influence on Cassandra. Dynamo uses consistent hashing to disperse data to the key-value store, and provides high adjustability. Consistent hashing sequences the value (slot) that hashed the key and placed it in a ring format. The multiple nodes that create the cluster processes certain ranges of the ring. Therefore, every time a node in the cluster falls out or comes in, the closed node on the ring can take over the concerned range or distribute the range without rehashing. Cassandra also raises the usability level with the concept of consistency level. This concept is related to replication. Confirmation of the number of replications and the completion of replication can be adjusted with the system parameter. For example, if three replications are maintained and a write operation is inserted, then the operation is only considered to be successful if the three replications are completed. However, Cassandra allows only N (under 3) number of executions to be checked and immediately returns to value. Therefore, write can be conducted successfully even with failures in the replication node, which raises usability. Histories of failed write operations are recorded on a different node, and the operation can be retried at a later date (this is called “hinted handoff”). Since the success of replicated writing is not guaranteed, the data suitability is checked in the reading stage. Generally, if there are multiple replications, it is collected into one when reading. However, Cassandra keeps in mind that not all replications match, and reads the data from all three replications, checks whether it is identical, and restores the latest data if it is not suitable (this is called “read repair”). This is why Cassandra is slower in reading than writing. HBase Now let’s look into HBase. HBase has the same structure as BigTable. While BigTable operates on the tablet unit, HBase disperses and replicates with the region unit. Key is arranged according to ranges, and the location of the region, where the key is stored in, is saved as meta data (this is called meta region). Therefore, if a key is inserted, first find the region, and later the client will cache this information. HBase places the writing operation in a single region, without distributing and splits the region if it becomes too large. Adjustments, such as creating regions in advance, must be made for the action above. HDFS is in charge of data storage and replication. When writing a file with Hadoop’s distribution file system, HDFS writes multiple replications (synchronous replication) and only reads one of them when reading. This guarantees data integrity and lowers reading overhead. However, since HBase and HDFS operates separately, the location of the node that contains memtable and SSTable may change, which may cause additional network I/O. What happens if there is a failure in the node? If the region server collapses, other nodes take over the tasks which comprised the region. In other words, the commit log that pertains to the region is replayed and memtable is created. If the failed node contains HDFS’s data, then chunk that pertains to the node is migrated to a different node. Thus, we have looked at the similarities and differences of Cassandra and HBase. Then why did the two products show a difference in writing during the comparison test? First of all, there is a difference in commit log. Cassandra writes the commit log once in the local file system. However, HBase enter the commit log in the HDFS and HDFS conducts the replication. Data locality is not put into consideration, so the file write request will probably be sent to a different physical node in the network. Second, Cassandra receives the write request on all three nodes, but HBase only ‘writes’ on a single region in the beginning, and receives requests on only one node. Then why is there a difference in reading performance? There are various differences, but for one READ request Cassandra reads the data three times while HBase only reads the data once. This places an influence on the throughput. MongoDB Architecture MongoDB is unlike Cassandra or HBase eplained earlier. There are too many functional groups that can be classified as NoSQL, which makes it difficult to compare with Cassandra and HBase on the same level. Cassandra and HBase is for processing full-scale large amounts of data, while MongoDB can be used quickly, schema-free when using a certain amount of data. MongoDB adopts a documented-oriented format, so it is more similar to RDBMS than a key-value or column oriented format. MongoDB operates on a memory base and places high performance above data scalability. If reading and writing is conducted within the usable memory, then high-performance is possible. However, performance is not guaranteed if operations exceed the given memory. Therefore, MongoDB should be used as a medium or small-sized storage system. Configure a test condition similar to the original test on a lower specification server and test 300 thousand records instead of 50 million, and MongoDB will record greater performance than Cassandra or HBase. Mongo DB uses BSON for data storage. (As known as Binary JSON, but is similar to Hessian than JSON in data encoding format and expressive datatype.) The logical unit that is expressed using BSON is called document. (When using NPC as an example, it is the same as when the NPC data is recorded on the disk and searched.) Document is a concept that corresponds to a row in RDBMS. The difference is that a document in MongoDB does not have a schema. Accumulated documents are called collections. A collection corresponds to a table in RDBMS. Basic query method is to search the collection to find documents that meet the conditions. Each document has a field called _id, which acts as the OID (Object Identifier), and all query results are returned with this _id. Index can also be created for collections. An index can be prepared for the fields in the document. This index is implemented in B-Tree, similar to that of RDBM’s index. Each collection creates an index for _id by default. One of the strengths of MongoDB is that its functions are similar to RDBMS, in that it supports indexing and JOIN. Further, we will now look at the structure of MongoDB. MongoDB Structure MongoDB is configurable in replication and distribution configuration. The system configuration recommended by MongoDB is three shard servers and three replica sets for each server (in other words, nine nodes). Replications are made possible by Master/Slave or a replica set. Replica set supports automatic failover. It has one master and multiple slaves, and write can only be sent to the master. If the master is terminated, service is resumed after one of the slaves are appointed as a master. The distribution method provided by MongoDB is sharding. Each collection can shard, and a field in the Collection can designate a shard key. Like HBase, sharding is operated with the Order-preserving partitioning method. Chunk is included in the , a continuous data range within shard, and can be as big as 64M. Chunks that have exceeded the maximum size can split into two chunks, and the divided chunks can migrate to different shard servers. The following figure contains a picture of MongoDB, which is composed of shards and replica sets. The client first contacts a process called mongos, which is in charge of routing and coordination. Coordination is done over multiple nodes through requests and the results are collected and returned. The config server retains meta information on shards and chunks. Now let’s look into MongoDB's internal structure. MongoDB uses the asynchronous method for replication. When a “write” operation is inserted, master stores the history in Oplog. Oplog is a fixed sized, FlfO method collection called capped collection. Logs accumulated in the Oplog are regularly polled by the slave, read, then implemented. If the slave failed to replay the log while the Oplog fills up, then the slave must suspend log-based replications and synchronizes by resyncing the master’s data. One of Oplog’s other distinctive characteristics is that it can conduct the saved operations idempotently when replaying the log. Operation logs are saved in the order it was executed in, so logs can be replayed after that period. Ops such as increment are changed to set ops when logged for this purpose. MongoDB’s replication and failover method and query power is similar to RDBMS. The most significant difference is that flexibility in data storage is provided to the application while schema-free data is stored. RDBMS Architecture Unlike NoSQL, RDBMS is "read" operation oriented. The detailed method differs for each product, but generally NoSQL products are implemented so that mass data can be ‘written’ quickly. On the other hand, RDBMS, which is ‘read’ oriented, not only enables quick reading, but also provides many functions and inquiry features. DBMS can quickly inquire values using indexes, but at the same time indexes slow down ‘writing’. Since an index must be configured for every ‘writing’ operation, the more indexes there are, the more time is needed to configure the indexes. The difference of RDBMS and NoSQL can be compared with ACID and BASE, but comparisons can also be made with reading and writing. If the business you are to build requires more reading than writing, and needs complex reading methods, as well as basic aggregate functions such as SUM() or AVG(), then RDBMS might be a better choice than NoSQL. Summary We have looked at benchmark tests done on Cassandra, HBase, MongoDB and the differences between the products. Cassandra and HBase show excellent writing abilities, but its reading performance did not meet expectations. This was because unlike existing RDBMSs, the two products were optimized for writing, and this resulted in a lot of concurrent I/O when reading. MongoDB has similar structures to that of RDBMS and shows great flexibility in data modeling especially for medium and small-sized businesses. By Lee Hye Jeong, Storage System Development Team, NHN Corporation. [Less]

I would like to announce the changes we have made regarding the version number of the next CUBRID release. First time when I wrote about the Roadmap for the next release, I mentioned that it was going to be CUBRID 3.2. The next version is expected ... [More] to introduce many new features and enhancements: 62 new SQL syntax extensions for better MySQL Compatibility Improved Index Enhancement for higher performance Easier High-Availability feature for more convenient and easy-to-use configurations This is why we decided to increment the minor version digit, instead of the maintenance number. So, the next version will be CUBRID 4.0 or the full version name - CUBRID 2008 R4.0, which will probably make in April this year. At this moment this is only a tentative forecast. For more information about the CUBRID Release Policy, visit http://www.cubrid.org/release_policy. [Less]

I would like to announce the changes we have made regarding the version number of the next CUBRID release. First time when I wrote about the Roadmap for the next release, I mentioned that it was going to be CUBRID 3.2. The next version is expected ... [More] to introduce many new features and enhancements: 62 new SQL syntax extensions for better MySQL Compatibility Improved Index Enhancement for higher performance Easier High-Availability feature for more convenient and easy-to-use configurations This is why we decided to increment the minor version digit, instead of the maintenance number. So, the next version will be CUBRID 4.0 or the full version name - CUBRID 2008 R4.0, which will probably make in April this year. At this moment this is only a tentative forecast. For more information about the CUBRID Release Policy, visit http://www.cubrid.org/release_policy. [Less]

Update: added the Distinguished clients for DotNetNuke. These days many websites (in fact, millions) are implemented by the use of popular open source content management systems (CMS). I would say, for everything you would imagine your site doing ... [More] , there is one or another CMS which can do that. Thus, the reason for choosing a particular one depends on the tasks you want to accomplish. For blogs there is a common sense of using WordPress blog management system, for large scale websites Drupal CMS is perceived to be a better fit, while Joomla CMS is easier to learn for newbies. .NET developers prefer alternative DonNetNuke or umbraco CMS. Besides these, CMS Made Simple and Liferay are amongst the most popular content management systems based on the number of the number of downloads. Elgg and MODx are amongst the rising stars. I will not tell you what a CMS is. If you want to learn, see the CMS Wikipedia page. In this blog I will give a comprehensive overview of top 14 content management systems based on the number of weekly downloads, installations, and brand familiarity. The full statistical data can be found in the 2010 OPEN SOURCE CMS MARKET SHARE REPORT by water&stone (2010). The report is distributed under the Creative Commons Attribution-Noncommercial License (3.0). Today we will cover Drupal, Joomla, MODx, WordPress, DotNetNuke, umbraco, Liferay, TYPO3, CMS Made Simple, MOVABLE TYPE, Plone, eZ Publish, concrete5, and Alfresco content management systems. You might already be familiar with some of these, while some can be new to you. I bet you will be surprised to learn that some unknown for you systems suit your requirements much better than those you have got to use so far (my personal experience). So, let's learn about each of them, and do not forget to leave your comments below on what other CMS should have been included in this list and why. Note: the order of the CMS does not represent its quality, performance, or superiority over the other. They are numbered to give a better perception to readers. The users feedback and opinions are retrieved from Stack Overflow, thus can be subjective as they are based on personal views and preferences. # 1: Drupal Official site: http://drupal.org/ 1% of all websites on the Internet are based on this platform. An estimated 7.2 million sites were powered by Drupal as of July 2010. Drupal started out in 2001. In one year from May 2007 to April 2008, Drupal was downloaded from their official site more than 1.4 million times, an increase of approximately 125% from the previous year. Weekly downloads: 33,671 (ranked #3 after Joomla and before DotNetNuke). Installations: 575 according to the survey (#3 after WordPres and before DotNetNuke), but 1.4% (#3 after Joomla and before Typo3) of Alexa Top 1 million sites. Brand Familiarity: Drupal is known to be the #3 most familiar content management system. Major Features: You can manage multiple sites with Drupal in multiple languages. You can use it for blogging site, corporate site, personal site, gallery, briefly, whatever you imagine. You can easily manage your site users, providing standard registration, including OpenID support. You can set various access control rules to limit the activity of your site users. Provides multiple-level menu system, template customization, advanced search, RSS feed aggregator. Officially Drupal supports several databases including MySQL, PostgreSQL, MariaDB, and SQLite. To increase its performance, you can use caching. At the same time it provides high security with notifications about the new update releases. Provides Search Engine Friendly descriptive URLs. Powered by jQuery JavaScript framework. Extensions: over 7,000 free community-contributed addons, known as contrib modules. Distinguished Clients: http://www.whitehouse.gov/ http://data.gov.uk/ http://www.ubuntu.com/ http://www.alrc.gov.au/ http://www.nysenate.gov/ http://london.gov.uk/ http://www.rutgers.edu/ http://www.economist.com/ http://wfp.org/ What users say: More difficult to master especially for newbies. It's more for advanced users. Though the new Drupal 7 claims to provide significantly improved usability (maybe more toward WordPress style). To achieve this, they hired web designers to specifically address the UX problems it had in previous versions. New major releases are not quite backward compatible. More focus on new features and functionality. External module developers should take care of compatibility themselves, except for data representation, which Drupal is intended to keep same. Drupal is often compared with Joomla and is perceived to be a bit slower. Though this depends on the type of website you develop. However, leveraging its caching and gzipping technologies, it is possible to achieve quite impressive results. jQuery, the default JavaScript framework in Drupal, allows to use alternative Mootools framework at the same time. # 2: Joomla Official site: http://www.joomla.org/ Joomla is the result of a fork of Mambo CMS on August 17, 2005. Within its first year of release, Joomla had been downloaded 2.5 million times. Weekly downloads: 113,836 (ranked #2 after WordPress and before Drupal). Installations: 1,297 according to the survey (#1 before WordPress), but 2.5% (#2 after WordPress and before Drupal) of Alexa Top 1 million sites. Brand Familiarity: Joomla is known to be the #1 most familiar content management system. Major Features: You can manage multiple site with Joomla in multiple languages natively (since Joomla 1.6). You can use it for blogging site, corporate site, personal site, gallery, briefly, whatever you imagine. You can easily manage your site users, providing standard registration, including Google OpenID support. Full support for Access Control List. Provides Multiple-level menu and content category system, template customization, advanced search, RSS feed aggregator. Officially supports only MySQL. Page cashing for increased performance. Provides moderate descriptive URLs (still not fully customizable as you can do in WordPress). Powered by MooTools JavaScript framework. Extensions: There are over 6,000 free and commercial plugins available from the official site. Distinguished Clients: http://www.linux.com/ http://www.itwire.com/ http://www.quizilla.com http://www.ihop.com http://gsas.harvard.edu Citibank (Financial institution intranet) - Not publicly accessible http://www.greenmaven.com http://www.outdoorphotographer.com http://www.playshakespeare.com http://www.sensointeriors.co.za What users say: More intuitive and easy to use than Drupal, though still not like WordPress. Powerful. Fully-fledged content management system, so you can create whatever site you want. Really strong security. If security problems found, immediately fixed. The new Joomla 1.6 release is expected to be faster, more convenient, with more features. Its strong dependency on Mootools JavaScript framework sometimes bothers users as Joomla does not give easy workaround to disable it and use jQuery instead. Support of only one database, limits Joomla a lot in terms of the number of users. However, this is a compromise for high optimizations for MySQL, thus increased overall performance. Does not allow to fully customize URLs - a must feature for CMS. #3: modx Official site: http://www.modxcms.com/ modx is not just an open source CMS but also a web application framework. Raymond Irving and Ryan Thrash began the MODx CMS project in 2004 as a fork of Etomite. In 2008 MODx users created a new logo and branding for the project. Now MODx allows for full segregation of content (plain HTML), appearance and behavior (standards compliant CSS and JavaScript) and logic (PHP, snippets). Weekly downloads: 4,500 (ranked #11 after umbraco and before Tiki). Installations: 58 according to the survey (#12 after eZ Publish and before umbraco), has less than 0.1% among the Alexa Top 1 million sites. Brand Familiarity: #14 (before Liferay and after eZ Publish). Major Features: As with Joomla, modx officially supports only MySQL database. Not just CMS but a PHP framework for Web. Freedom to choose jQuery, Mootools, ExtJS, Prototype or any other JavaScript library. Supports PHP 4.3.11 and above. Complete control of all metadata and URL structure for SEO (Search Engine Optimization). Unlimited hierarchical page depth. Can create custom fields and widgets for templates. Role-based permissions for the Manager. Ability to customize the Manager on a per-deployment basis. Ecommerce integration via Foxy Cart. Extensions: 622, also known as add-ons. Distinguished Clients: http://www.pippatoledo.com/ http://www.aquevix.com/ http://www.not1bug.com/ http://everlight-uva.com/ http://www.tritopora.ru/ http://www.strategische-webloesungen.de/ What users say: Good to have a choice for favorite JavaScript framework. Light CMS solution (but not necessarily the fastest). PHP 4 support for developers mean that a lot of compromises had to be made in terms of OOP (Object Oriented Programming) in order to offer PHP 4 support . Nice to have freedom to set custom URL. #4: WordPress Official site: http://wordpress.org/ WordPress was first released on May 27, 2003, by Matt Mullenweg as a fork of b2/cafelog. As of August 2010, version 3.0 had been downloaded over 12.5 million times. Nowadays, known as the #1 CMS for blogging. Weekly downloads: 983,625 (ranked #1 before Joomla). Installations: 1,012 according to the survey (#2 after Joomla and before Drupal), but 12.9% of the Alexa Top 1 million sites (#1 before Joomla). Brand Familiarity: #2 (after Joomla and before Drupal). Major Features: Highly optimized for blogging. Custom and easy to switch themes. Users can re-arrange widgets without editing PHP or HTML code. Official support for only MySQL. Custom URL, clean permalink structure, excellent for SEO. Nested, multiple categories to articles. Support for tagging. Advanced search by tags. Highly intuitive UI (User Interface). jQuery JavaScript framework. Supports the Trackback and Pingback standards for displaying links to other sites that have themselves linked to a post or article. Rich plugin architecture which allows users and developers to extend its functionality beyond the features that come as part of the base install. Native applications exist for Android, iPhone/iPod Touch, and BlackBerry which provide access to some of the features in the WordPress Admin panel and work with WordPress.com and many WordPress.org blogs. Extensions: 12,780 plugins and 1,315 themes. Distinguished Clients: http://www.nytimes.com/interactive/blogs/directory.html http://tmagazine.blogs.nytimes.com/ http://crowdfavorite.com/ http://blog.broadband.gov/ http://blogs.america.gov/ http://www.number10.gov.uk/ http://www.speaker.gov/ http://allthingsd.com/ http://politicalticker.blogs.cnn.com/ http://stylenews.peoplestylewatch.com/ What users say: Perhaps, the most convenient, easy to use and intuitive CMS (or BMS) in the world. Perfect for blog sites. But if you need to develop a dynamic site with various components, perhaps, other CMS would fit better, though, WordPress provides enough plugins to accomplish almost all tasks. I would avoid Wordpress as a CMS in a professional environment. As stated earlier, it's a great blogging platform, but doesn't generally offer the robustness that most professional environments require. Availability of jQuery makes the plugin development a lot easier for external developers and site owners. Endless themes - no need to worry about the new design for your site, unless you really need something special. #5: DotNetNuke Official site: http://www.dotnetnuke.com/ DotNetNuke is an open source platform for building web sites based on Microsoft .NET technology. It is written in VB.NET and distributed under both a Community Edition BSD-style license and a commercial proprietary license. The Community Edition is a popular web content management (WCM) system and application development framework for ASP.NET, with over 6 million downloads and 600,000 production web sites as of October 2010. More than 8,000 DotNetNuke apps are available for purchase on Snowcovered.com. DotNetNuke.com has over 800,000 registered members as of October 2010. Weekly downloads: 13,000 (ranked #4 after Drupal and before CMS Made Simple). Installations: 402 according to the survey (#4 after Drupal and before Liferay), but 0.2% of the Alexa Top 1 million sites (#4 after Typo3 and before MOVABLE TYPE). Brand Familiarity: #4 (after Drupal and before Typo3). Major Features: Distinguishes between community (common features) and enterprise (full set of features) editions. Various modules, and data providers. Provides language packs for about 60 languages. Customizable through skins and templates. Distinguished clients: http://www.chamberlain.edu/ http://magenic.com/ http://www.graphiksolutions.ca/ http://www.marly.com/ http://www.dotcomsoftwaresolutions.com http://www.cityplaceascent.com/ http://sites.kiwanis.org/kiwanis/en/home.aspx http://www.zonediet.com/ What users say: A little bit difficult to create modules. It is perceived as a little bit bulky CMS. Does not provide extensive documentation and user guides. Similar to most CMS, does not provide full backward compatibility in its new major releases. Unlike its enterprise edition, the community edition is not tested and certified by the DotNetNuke Corporation. Even if it provides the language packs, the sites cannot be created to support multiple languages. Third party module should be used to enable this feature. Auto-upgrade, Advanced Site Search, Page Cashing, and many must-have features are not included in the Community edition, only in Professional or Enterprise editions. #6: Umbraco Official site: http://umbraco.org/ Umbraco is also an open source content management system. It was developed by Niels Hartvig in 2000 and released as open source software in 2004. It is written in C# and can be deployed on Microsoft based infrastructure. In 2010, with about 1,000 downloads a day, Umbraco was in the Top 5 most popular downloads via the Microsoft Web Platform Installer, two places below its main rival DotNetNuke. Weekly downloads: 5,420 (ranked #10 after Alfresco and before MODx). Installations: 57 according to the survey (#13 after MODx and before e107), but less than 0.1% of the Alexa Top 1 million sites. Brand Familiarity: #16 (after Liferay and before e107). Major Features: Can be deployed with several databases, including MySQL, SQL Server, and VistaDB. SEO-friendly URLs. Extensions: 310 add-on modules. What users say: Limited number of extensions. Official support for Windows OS only. Umbraco is oriented to small low-cost sites. Video trainings have to be purchased. Many must-have features have to purchased. #7: Liferay Official site: http://www.liferay.com/ Liferay Portal is a free and open source enterprise portal written in Java and distributed under the GNU Lesser General Public License. It allows users to set up features common to websites. It is fundamentally constructed of functional units called portlets. Liferay is sometimes described as a content management framework or a web application framework. It comes with certain portlets preinstalled. These comprise the core functionality of the portal system. Weekly downloads: 9,435 (ranked #6 after CMS Made Simple and before TYPO3). Installations: 154 according to the survey (#5 after DotNetNuke and before TYPO3), but less than 0.1% of the Alexa Top 1 million sites. Brand Familiarity: #15 (after MODx and before Umbraco). Major Features/ Portlets: Can tag and categorize contents. Document Library Manager, Recent Documents. Alfresco, Documentum, and other document library integration. User management based on various roles and groups (ACL). WebDAV Integration (Web-based Distributed Authoring and Versioning which allows users to collaboratively edit and manage files on remote web servers). Nested Portlets User Directory LDAP Integration Microsoft Office Integration Calendar/Chat/Mail/Message Boards/Polls Wiki (supports Creole as well as MediaWiki syntax) Alerts and Announcements Knowledge Base Social Equity Can create multi-language sites. Asset Publisher to publish many contents, tagged by a specific term, at once. Extensions: 27 official plugins and 208 community developed plugins. Distinguished Clients: It is primarily used to power corporate business sites. http://developer.cisco.com http://www.t-mobile.cz http://www.betavine.net http://www.foxchannel.de/ http://www.ixarm.com What users say: It is mostly used by enterprise companies rather than for powering personal or community sites, though it has social and collaboration features. More professional developers driven project (backed by Liferay Inc.) rather than the community driven. Provide paid Enterprise edition. All features are available in Community edition (except for support and customization related). #8: Typo3 Official site: http://typo3.org/ TYPO3 is a free and open source CMS released under the GNU General Public License oriented to small to mid size enterprise-class users. TemplaVoila is an alternative template engine extension for TYPO3. A graphical mapping tool for creating templates is included, an alternative page module, the ability to create flexible content elements and an API for developers. New content element types can be created without programming. TemplaVoila facilitates more flexibility for maintaining web pages than TYPO3's standard templating, while making it possible to enforce a strict corporate design and allowing editors to work with content more intuitively. Weekly downloads: 7,461 (ranked #7 after Liferay and before eZ Publish). Installations: 122 according to the survey (#6 after Liferay and before Tiki), 0.6% of the Alexa Top 1 million sites (#4 after Drupal and before DotNetNuke). Brand Familiarity: #5 (after DotNetNuke and before OpenCMS). Major Features: Supports MySQL, Oracle, MS-SQL, PostgreSQL, ODBC, LDAP - virtually any external data source. You can undo any change you make on the site. Can create multiple sites with multiple domains for each. Can have multiple template per site. User management. Able to switch from administrator user to general user to check permissions. Sandbox: administrators can set up a section within the system to test new features without disturbing the main site. Versioning of content pages. Advanced caching: template, navigation or page level. Link management. Multi-language content. Search Engine friendly URLs. Extensions: more than 4,500 pluggable extensions are available for TYPO3. Distinguished clients: http://www.volkswagen-ir.de http://www.eadssecureuk.com http://www.geveriwise-eu.com http://www.rewe-xxl.de What users say: The administrator's UI (User Interface) is not user-friendly, though intuitive and functional. It is extremely powerful which provide more enterprise level features, but the learning curve is incredibly steep. Great ease of multi-lingual site management. Difficult to customize templates. Need to learn TypoScript and TemplaVoila, two TYPO3-specific systems. #9: CMS Made Simple Official site: http://www.cmsmadesimple.org/ CMS Made Simple is an open source cms built using PHP with support for MySQL and PostgreSQL. The template system is driven using the Smarty Template Engine. Weekly downloads: 9,948 (ranked #5 after DotNetNuke and before Liferay). Installations: 72 according to the survey (#7 after Tiki and before Alfresco), 0.1% of the Alexa Top 1 million sites (#8 after Xoops and before eZ Publish). Brand Familiarity: #12 (after Xoop and before Ez Publish). Major Features: Officially supports MySQL and PostgreSQL. Search Engine Friendly URLs. Users management and group based permissions system. Content tagging. Site localization is available in 20 languages. Extensions: unknown, but many. What users say: It is oriented more toward professional users. It does not provide many templates, so coding knowledge is a must. #10: Movable Type Official site: http://movabletype.org/ Movable Type is a weblog publishing system, similar to WordPress, developed by the Six Apart company in Perl programming language. At various times, this company maintained three other publishing systems - TypePad, Vox, and LiveJournal. Movable Type was publicly announced on September 3, 2001. Version 1.0 was publicly released on October 8, 2001, thus it is a blogging system older than WordPress. On 12 December 2007, Movable Type was relicensed as free software under the GNU General Public License. Based on the list of its customers, Movable Type is quite credible CMS. Weekly downloads: unavailable. Installations: 30 according to the survey (#17 after Silverstripe and before OpenCMS), 0.1% of the Alexa Top 1 million sites (#6 after DotNetNuke and before Xoops). Brand Familiarity: #8 (after Plone and before Alfresco). Major Features: Convenient blogging system with social community features. Since version 5 officially supports only MySQL. PostgreSQL and SQLite can be used via plugins. Databases such as Oracle can be integrated with Movable Type Enterprise edition. Manage user roles. OpenID support. Multiple site hosting. Easily customizable templates. Revision history. Can add custom fields. Content tags and categories. Feeds and trackback links. Available localization and internationalization. Can generate static pages (updated whenever the content of the site is changed). Extensions: about 1,000 plugins. Distinguished clients: http://www.britneyspears.com/ http://www.barackobama.com/ http://boeingblogs.com/ http://blogs.oracle.com/ http://gehealthcare.typepad.com/ http://www.spd.org/ http://blog.ted.com/ What users say: Users often deploy Movable Type for their blog sites, newspaper or other type of publishing sites. A powerful alternative for WordPress. #11: Plone Official site: http://plone.org/ Plone, a free and open source CMS, started in 1999 by Alexander Limi, Alan Runyan, and Vidar Andersen. It was made as a usability layer on top of the Zope content management framework, thus Plone is written in Python. The first version was released in 2001. In 2004, Plone 2.0 was released. This release brought more customizable features to Plone, and enhanced the add-on functions. In 2007, Plone 3 was released. This new release brought inline editing, an upgraded visual editor, and strengthened security, among many other enhancements. Recently in 2010, Plone 4 was released with major improvements in performance. Weekly downloads: unavailabe. Installations: 34 according to the survey (#15 after e107 and before Silverstripe), 0.1% of the Alexa Top 1 million sites (#10 after eZ Publish). Brand Familiarity: #7 (after OpenCMS and before Movable Type). Major Features: Inline editing - no need to reload the page for editing. Localized into 40 languages. Plone can be integrated with Active Directory, Salesforce, LDAP, SQL, Web Services, and Oracle. Working Copy support - keep the old version of a content published until you publish a new version. Cut/copy/paste operations on content. Link and reference integrity checking - no more broken links within your site. Can create workflows - useful for organizations. LiveSearch - instant site search powered by AJAX. Full-text indexing of Word and PDF documents. Wiki support Collaboration and sharing Automatic locking and unlocking Versioning, history and reverting content Authentication back-end Collections Multilingual content management Automatic previous/next navigation Human-readable URLs Caching proxy integration Drag and drop reordering of content Adjustable templates on content RSS feed support Automatic image scaling and thumbnail generation Comment capabilities on any content WebDAV and FTP support Hot -backup support Extensions: 1490 add-ons. Distinguished clients: http://www.amnesty.ch/en http://www.brasil.gov.br/ http://www.chicagohistory.org/ http://ccnmtl.columbia.edu/ http://cnx.org/ http://discovermagazine.com/ http://www.ece.rice.edu/ http://www.engagemedia.org/ http://science.nasa.gov/ What users say: It has a lot of features (several times more than listed above) built-in. To become a Plone expert is long and expensive. It has a steep learning curve. Plone is backed by Zope framework which is very powerful with support for caching, rollback, etc. - everything what your organization might need. Plone is really complex to deeply tweak. Plone is known for its high security. Plone runs slow if you don't know how to optimize it. #12: eZ Publish Official site: http://ez.no/ eZ Publish is an open source enterprise CMS developed by the Norwegian company eZ Systems in 1999 using PHP programming language. eZ Publish is freely available under the GPL licence, as well as under proprietary licenses that include commercial support. eZ Publish supports the development of customized web applications. Typical applications range from a personal homepage to a multilingual corporate website, which include role-based multi-user access, e-commerce functions and online communities. Weekly downloads: 7,031 (ranked #8 after TYPO3 and before Alfresco). Installations: 60 according to the survey (#11 after Concrete5 and before MODx), 0.1% of the Alexa Top 1 million sites (#9 after CMS Made Simple and before Plone). Brand Familiarity: #13 (after CMS Made Simple and before MODx). Major Features: Advanced Search feature is available in only Enterprise Edition. Online image editor. Can create building blocks for the site and later reuse them in several pages. Distinguished clients: http://www.schemexpert.com/ http://www.ticotimes.net/ http://jp.wsj.com/ http://www.laborange.fr/ http://canalstreet.canalplus.fr/ http://www.hks.harvard.edu/ http://www.vogue.com.au/ What users say: The documentation is a bit limited especially there is almost no documentation for module development. #13: Concrete 5 Official site: http://www.concrete5.org/ Concrete5 is an open source CMS started in 2003 as a rapid-design approach to building the now-defunct LewisAndClark200.org, the official site for the Ad Council's National Council for the Lewis & Clark Bicentennial. Concrete5 is developed in PHP and is distributed under MIT software license. Weekly downloads: unavailable. Installations: 62 according to the survey (#10 after Alfresco and before eZ Publish), has less than 0.1% of the Alexa Top 1 million sites. Brand Familiarity: #20 (after TextPattern). Major Features: Integrated server side caching. Support for only MySQL. Inline content editing. Image editing tool. Editable areas are defined in concrete5 templates which allow editors to insert 'blocks' of content. Additional blocks are available as add-ons. Automatic upgrade is available. Advanced Permissions to track content versions. Distinguished clients: http://www.genco.com http://www.cottonfrombluetogreen.org/ http://www.cs.uh.edu/ http://www.signals.ca/ http://www.paulraymondgregory.com What users say: It's PHP based and quite new but it's quite a nice layout and it's really natural for new CMS users. You can go from a paper-based sitemap and PSD to a full site structure, ready for data entry, within a day, two at a push. Concrete5 is simple, suitable for creating sites quickly. Creating template is very easy with Concrete5. #14: Alfresco Official site: http://www.alfresco.com/ Alfresco is an open source enterprise content management system for Microsoft Windows and Unix-like operating systems. Alfresco includes a content repository, an out-of-the-box web portal framework for managing and using standard portal content, a CIFS interface that provides file system compatibility on Microsoft Windows and Unix-like operating systems, a web content management system capable of virtualizing web apps and static sites via Apache Tomcat, Lucene indexing, and jBPM workflow. The Alfresco system is developed using Java technology. John Newton (co-founder of Documentum) and John Powell (a former COO of Business Objects) founded Alfresco Software, Inc. in 2005. Weekly downloads: 7,000 (ranked #9 after Ez Publish and before Umbraco). Installations: 70 according to the survey (#9 after CMS Made Simple and before Concrete5) Alexa ranking is not available. Brand Familiarity: #9 (after Movable Type and before Tiki). Major Features: Document Management. Web Content Management (including full webapp & session virtualization). Repository-level versioning (similar to Subversion). Records Management, including 5015.2 certification. Repository access via CIFS /SMB, FTP, Web DAV, NFS and CMIS. j BPM workflow. Advanced search with Lucene. Multi-language support. Officially runs on Windows, Linux and Solaris. User Interface official supports Internet Explorer and Firefox. Desktop integration with Microsoft Office and OpenOffice.org. Clustering support. Pluggable authentication: NTLM, LDAP, Kerberos, CAS. Distinguished clients: no links are available but numerous case studies can be found on Alfresco home page. France Air Force Harvard Business School Publishing Toyota Sony Pictures Fox National Academy of Sciences Cisco What users say: Alfresco is mostly for enterprises rather than for personal sites. Simple to install and use, flexible and open-ended. Alfresco is a solution with the broadest range of technical capabilities and the best feedback from users. In addition to demonstrating a promising roadmap for collaboration tools, Alfresco was highly attractive from a cost perspective, compared to the proprietary products offered by other ECM vendors. All in one solution for enterprises. See also Besides, the official home pages, you can refer to Wikipedia for general information on these content management systems. Also Stack Overflow provides very informative users feedbacks on each of these CMS. http://en.wikipedia.org/wiki/Drupal http://en.wikipedia.org/wiki/Joomla http://en.wikipedia.org/wiki/Modx http://en.wikipedia.org/wiki/Wordpress http://en.wikipedia.org/wiki/DotNetNuke http://en.wikipedia.org/wiki/Umbraco http://en.wikipedia.org/wiki/Liferay http://en.wikipedia.org/wiki/Typo3 http://en.wikipedia.org/wiki/CMS_Made_Simple http://en.wikipedia.org/wiki/Movable_Type http://en.wikipedia.org/wiki/Plone_(software) http://en.wikipedia.org/wiki/EZ_Publish http://en.wikipedia.org/wiki/Concrete5 http://en.wikipedia.org/wiki/Alfresco_(software) http://www.waterandstone.com/sites/default/files/2010%20OSCMS%20Report.pdf http://php.opensourcecms.com/ http://stackoverflow.com/ [Less]

Twitter - one of the latest and hottest Web 2.0 trends used by millions of users around the world every day. How does it manage the enormous data produced by its users? This article reviews the technical aspects of the Twitter service, how it handles ... [More] such a tremendous amount of tweets and other data, and looks at what we can learn from the findings. Twitter As we all know, Twitter, which launched its service in 2006, is expanding at an amazing pace. According to official About Twitter page, as of September 2010 some 175 million users use the service, generating about 95 million tweets a day. This means about 1100 new tweets appear every second. At peak times, over 3,000 tweets are generated per second. More than 600 million searches are done in a day. Simply put, the scale of service is amazing, but what is even more amazing is its growth rate. Behind Twitter's rapid growth is the seamless processing of huge amounts of data. Understanding Twitter For those of us who are satisfied with sitting in the passenger seat, here is a brief outline of the services provided by Twitter. Twitter is a micro blogging service. Users are limited to sending 140 characters per message, which is called a tweet. Here are two core services of Twitter: One is to allow you to read (follow) the tweets of others in whom you are interested. The other is to allow other users (followers) who are interested in you to read your tweets. The other services provide the RT function (ReTweet, or posting a link to another user's tweet), and the Reply function, which lets you reply to tweets of others. Replied tweets can only be seen by users who follow both you and the owner of the original tweet. These two additional features are what make it difficult to implement Twitter services and complicate the matters of designing systems as well as creating headaches to the administrators. Just like G-dragon is the most popular user on me2Day, Korea's most popular alternative for Twitter, Lady GaGa (http://twitter.com/Ladygaga), a pop singer, has the largest number of followers on Twitter. As of today, 7,617,063 people follow Lady Gaga. In return for this popularity, Lady Gaga also follows 152,130 users. Katy Perry (http://twitter.com/katyperry), who is on the 8th position among the users with largest number of followers on Twitter, has 5,158,745 followers. Imagine the number of ReTweets and Replies generated by these two examples alone, and one can easily guess the devastating combination it would result in. When this passes the point of no return, one will see Twitter's Fail Whale. Real-Time Data in Twitter The data tweeted by users must be processed in real time. As mentioned earlier, it is not easy to attain low latency and high throughput at the same time when the size of the generated data is enormous. Here are four types of Real-Time Data that give Twitter engineers a headache: Tweets Timelines Social graphs Search indices Let's see how these four types of data are handled, and how the associated problems are solved. Tweets The typical value of the Tweet meta information consists of an id of a tweet and the id (user_id) of a user who posted it (in addition to the Reply information). The [single] RDBMS table which was used by earlier versions of Twitter is shown below: id user_id text created_at 20 12 just setting up my twitter 2006-03-21 20:50:14 29 12 inviting coworkers 2006-03-21 21:02:56 34 16 Oh shit, I just twittered a little. 2006-03-21 21:08:09 In this original implementation Twitter used Master-slave replication and Memcached for read throughput. But when Twitter became popular and the disk usage surpassed 800 GB, reaching a limit in terms of performance, the Table Partition became necessary. In this situation, there are two types of easily applicable partitioning methods. The user id-based partitioning The tweet id-based partitioning The first type has a drawback that N number of partitions must be searched for to find a specific tweet. The second case has a similar drawback which requires to search for N number of partitions to find the tweets created by a specific user. To circumvent these drawbacks, Twitter uses the time axis-based partitioning. It other words, this is a policy in which tweets are stored in chronological order and passed to the next partition when a table exceeds a certain size. This is reasonable, as most of the reading operations are for recent tweets. When searching for tweets created by a specific user, it is thus only necessary to search for a couple of the most recent partitions to retrieve a sufficient number of tweets. However, there is a problem with this method. In this solution, recent partitions are bound to become overloaded. To solve this problem, Twitter adds temporary partitions to recent ones in order to increase concurrency, and exclusively uses Memcached to increase the effectiveness of caching. This is what Twitter currently does to store tweets. The introduction of Cassandra, a NoSQL, is being considered for more effective handling of a larger number of tweets in the future. To reduce the burden of partitioning, Twitter is considering separating stores that take id as the key from stores, in which the sequence of tweet ids for each user (user_id) is the key. Twitter wants its tweet storages to be non-relational. (Unfortunately, this is not yet certain. The reason for this, I think, is that Cassandra's performance will sharply decrease when one of the nodes in a distributed data system will fail, causing reconfiguration which will increase the workload for other nodes). Timelines A Timeline is the tweet sequence of users who I follow. Considering the definition, a query to retrieve the 20 most recent tweets that a specific user will read might look something like this: SELECT * FROM tweets WHERE user_id IN  (SELECT source_id FROM followers WHERE destination_id = ?) ORDER BY created_at DESC LIMIT 20 The query above will show severely degraded service performance as I follow more users. Judging that such data schema and query cannot solve the problem, Twitter instead creates Timeline in advance through a process called Fanout (a kind of unfolding). In this way, the sequence of a new tweet from a user is created in advance by using the Fanout process. The key factor in this case is the processing speed of Fanout. The sequence is stored in Memcached in order to quickly apply new tweets to Timeline (Off-line fanout is a batch process that creates Timeline in advance by reflecting a new tweet regardless of user requests. On-line fanout, on the other hand, creates Timeline upon the request of a user.) Twitter has achieved an impressive boost in performance after the implementation of the fanout process. date average tps peak tps deliveries 2008-10-07 30 120 21,000 2010-04-15 700 2,000 1,200,000 The throughput has been improved from 21,000 Timeline requests per second to 1.2 million Timeline requests per second. The point in this example is that High Throughput and Low Latency can be achieved by changing the hierarchy of storage and the time at which operation takes place. That is, the memory and disk need to be utilized in a manner appropriate to the purpose, and both the off-line and online processes need to be used. Social Graphs Social Graphs is an expression of the relationship between a Follow and a Follower. The Following block may be added here, if we were to take it one step further. A variety of operations are required for this relationship, such as the union, intersection, a difference, and inclusion. As mentioned in the previous section, it is not that simple when it comes to Reply, as it is necessary to display the original poster of the tweet and the author of the Reply to other users as well. Source_id Destination_id 20 12 29 12 34 16 The initial format looked like the above. The relationship of Follow was stored in a single table and replicated in master-slave fashion. The problem with this, however, is that throughput for writing is unsatisfactory, and indices sometimes are not loaded into the memory as the size of the table grows. A solution for this is to maintain two tables – Forward and Backward. In this solution, each table is maintained according to the directions of the source and destination, and is partitioned by user_id. Forward Table source_id destination_id updated_at X 20 12 20:50:14 X 20 13 20:51:32 20 16 Backward Table destination_id source_id updated_at X 12 20 20:50:14 X 12 32 20:51:32 12 16 Using this method enables primary key-based partitioning, while speeding up the process and making it easier to manage indices. One more important thing: for effective partitioning and replication of a table, the data should be commutative (the order of commutative and operation is interchangeable) and idempotent (the result is the same when the operation is performed several times). This precondition is made by Gizzard, Twitter's powerful sharding (or partitioning) framework. In this case an operation that has failed to be executed due to a temporary malfunction can be executed again at any time later, regardless of whether it has been executed or not. To create an internally commutative attribute, all rows have a time stamp to ensure their validity, and the updateable attribute information in a relationship becomes invalid when the time recorded in its time stamp is earlier than the time recorded in the time stamp of the row. The performance of the Social Graphs of Twitter is as follows: cardinality iteration write materialize inclusion 1ms 100 edges/ms 16 ms 1 ms Search Indices Twitter supports a search function for specific keywords, as shown above. For this, it assigns an id to every possible keyword and stores the doc_id information that includes the keyword to provide the search function. term_id doc_id 20 12 20 86 34 16 In earlier versions of Twitter, the relationship between term_id and doc_id used to be stored in a single table, as shown above. But doing this increased the size of indices, to the point that they were too big to be loaded into the RAM. Partitioning was required to solve this problem, and thus a time-based partitioning was applied, which is similar to the partitioning of tweets. Doing this has the advantage of allowing the information to be immutable, which means that the data will not be changed once it has been created. In addition, it is a quite effective partitioning as it satisfies the commutative and idempotent attributes, both of which greatly help the replication and distribution processes. Twitter plans to further improve this feature by separating partitions for document and time and replacing MySQL, the lowest-level storage, with Lucene, which is optimized for searching. Twitter Performance The following is a table showing the performance data of the current Twitter service. reads/second writes/second cardinality Tweets 100 k 850 12 b Timelines 80 k 1.2 m a lot Graphs 100 k 20 k 13 b Search 13 k 21 k 315 m The Lessons Learned Here is what we can take out of this Twitter DB decomposition by analyzing its four major types of data. There are no permanent solutions (respond flexibly to changing circumstances). There are no perfect solutions - there are only solutions that are useful for a certain amount of time. Scalability does not achieve itself - it is achieved by appropriately combining partitioning, indexing, and replication. All data related to a Read-time query must be serviced in the memory (A disk is only for permanent storage). Only a small number of problems can be solved through pre-computation. For best performance always pay attention to locality! In Conclusion We have studied Twitter's approach of throwing a stone to catch two rabbits, namely Low Latency and High Throughput. Quick readers may have already noticed that they are chasing two hares at once by creating an additional attribute that NoSQL pursues while still using a RDBMS, as the lowest-level storage. In short, Twitter assigns the commutative and idempotent attributes to data for effective partitioning and replication. From perspective, Twitter is already NoSQL-ready (It remains to be seen whether that NoSQL is Cassandra). I have learned much from Twitter while looking for the Data Store for NHN. In addition, when I was asked to write a feature article about NoSQL by The Platform Magazine, I thought it would be beneficial for people with the same concerns to share this piece of information, as doing so might reduce the time they would spend on this issue. N-Store (a tentative name), the first high-speed storage system of NHN in the future, is under development. The N-Store project members are working hard to come up with a storage system that will live up to expectations. Kim Hyeo, Development Center, NHN. If you like this article, add me on http://twitter.com/cubrid. See also http://www.slideshare.net/netik/billions-of-hits-scaling-twitter http://twitter.com/about http://engineering.twitter.com/2010/05/introducing-flockdb.html http://en.wikipedia.org/wiki/Gizzard_(scala_framework) http://gigaom.com/2010/04/07/gizzard-anyone-twitter-offers-up-code-for-distributed-data/ http://github.com/twitter/gizzard https://github.com/twitter/flockdb https://github.com/twitter/gizzmo https://github.com/robey/kestrel http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster http://www.readwriteweb.com/cloud/2011/01/how-twitter-uses-nosql.php Big Data in Real-Time at Twitter http://www.slideshare.net/nkallen/q-con-3770885 (PDF link here) [Less]

Twitter - one of the latest and hottest Web 2.0 trends used by millions of users around the world every day. How does it manage the enormous data produced by its users? This article reviews the technical aspects of the Twitter service, how it handles ... [More] such a tremendous amount of tweets and other data, and looks at what we can learn from the findings. Twitter As we all know, Twitter, which launched its service in 2006, is expanding at an amazing pace. According to official About Twitter page, as of September 2010 some 175 million users use the service, generating about 95 million tweets a day. This means about 1100 new tweets appear every second. At peak times, over 3,000 tweets are generated per second. More than 600 million searches are done in a day. Simply put, the scale of service is amazing, but what is even more amazing is its growth rate. Behind Twitter's rapid growth is the seamless processing of huge amounts of data. Understanding Twitter For those of us who are satisfied with sitting in the passenger seat, here is a brief outline of the services provided by Twitter. Twitter is a micro blogging service. Users are limited to sending 140 characters per message, which is called a tweet. Here are two core services of Twitter: One is to allow you to read (follow) the tweets of others in whom you are interested. The other is to allow other users (followers) who are interested in you to read your tweets. The other services provide the RT function (ReTweet, or posting a link to another user's tweet), and the Reply function, which lets you reply to tweets of others. Replied tweets can only be seen by users who follow both you and the owner of the original tweet. These two additional features are what make it difficult to implement Twitter services and complicate the matters of designing systems as well as creating headaches to the administrators. Just like G-dragon is the most popular user on me2Day, Korea's most popular alternative for Twitter, Lady GaGa (http://twitter.com/Ladygaga), a pop singer, has the largest number of followers on Twitter. As of today, 7,617,063 people follow Lady Gaga. In return for this popularity, Lady Gaga also follows 152,130 users. Katy Perry (http://twitter.com/katyperry), who is on the 8th position among the users with largest number of followers on Twitter, has 5,158,745 followers. Imagine the number of ReTweets and Replies generated by these two examples alone, and one can easily guess the devastating combination it would result in. When this passes the point of no return, one will see Twitter's Fail Whale. Real-Time Data in Twitter The data tweeted by users must be processed in real time. As mentioned earlier, it is not easy to attain low latency and high throughput at the same time when the size of the generated data is enormous. Here are four types of Real-Time Data that give Twitter engineers a headache: Tweets Timelines Social graphs Search indices Let's see how these four types of data are handled, and how the associated problems are solved. Tweets The typical value of the Tweet meta information consists of an id of a tweet and the id (user_id) of a user who posted it (in addition to the Reply information). The [single] RDBMS table which was used by earlier versions of Twitter is shown below: id user_id text created_at 20 12 just setting up my twitter 2006-03-21 20:50:14 29 12 inviting coworkers 2006-03-21 21:02:56 34 16 Oh shit, I just twittered a little. 2006-03-21 21:08:09 In this original implementation Twitter used Master-slave replication and Memcached for read throughput. But when Twitter became popular and the disk usage surpassed 800 GB, reaching a limit in terms of performance, the Table Partition became necessary. In this situation, there are two types of easily applicable partitioning methods. The user id-based partitioning The tweet id-based partitioning The first type has a drawback that N number of partitions must be searched for to find a specific tweet. The second case has a similar drawback which requires to search for N number of partitions to find the tweets created by a specific user. To circumvent these drawbacks, Twitter uses the time axis-based partitioning. It other words, this is a policy in which tweets are stored in chronological order and passed to the next partition when a table exceeds a certain size. This is reasonable, as most of the reading operations are for recent tweets. When searching for tweets created by a specific user, it is thus only necessary to search for a couple of the most recent partitions to retrieve a sufficient number of tweets. However, there is a problem with this method. In this solution, recent partitions are bound to become overloaded. To solve this problem, Twitter adds temporary partitions to recent ones in order to increase concurrency, and exclusively uses Memcached to increase the effectiveness of caching. This is what Twitter currently does to store tweets. The introduction of Cassandra, a NoSQL, is being considered for more effective handling of a larger number of tweets in the future. To reduce the burden of partitioning, Twitter is considering separating stores that take id as the key from stores, in which the sequence of tweet ids for each user (user_id) is the key. Twitter wants its tweet storages to be non-relational. (Unfortunately, this is not yet certain. The reason for this, I think, is that Cassandra's performance will sharply decrease when one of the nodes in a distributed data system will fail, causing reconfiguration which will increase the workload for other nodes). Timelines A Timeline is the tweet sequence of users who I follow. Considering the definition, a query to retrieve the 20 most recent tweets that a specific user will read might look something like this: SELECT * FROM tweets WHERE user_id IN  (SELECT source_id FROM followers WHERE destination_id = ?) ORDER BY created_at DESC LIMIT 20 The query above will show severely degraded service performance as I follow more users. Judging that such data schema and query cannot solve the problem, Twitter instead creates Timeline in advance through a process called Fanout (a kind of unfolding). In this way, the sequence of a new tweet from a user is created in advance by using the Fanout process. The key factor in this case is the processing speed of Fanout. The sequence is stored in Memcached in order to quickly apply new tweets to Timeline (Off-line fanout is a batch process that creates Timeline in advance by reflecting a new tweet regardless of user requests. On-line fanout, on the other hand, creates Timeline upon the request of a user.) Twitter has achieved an impressive boost in performance after the implementation of the fanout process. date average tps peak tps deliveries 2008-10-07 30 120 21,000 2010-04-15 700 2,000 1,200,000 The throughput has been improved from 21,000 Timeline requests per second to 1.2 million Timeline requests per second. The point in this example is that High Throughput and Low Latency can be achieved by changing the hierarchy of storage and the time at which operation takes place. That is, the memory and disk need to be utilized in a manner appropriate to the purpose, and both the off-line and online processes need to be used. Social Graphs Social Graphs is an expression of the relationship between a Follow and a Follower. The Following block may be added here, if we were to take it one step further. A variety of operations are required for this relationship, such as the union, intersection, a difference, and inclusion. As mentioned in the previous section, it is not that simple when it comes to Reply, as it is necessary to display the original poster of the tweet and the author of the Reply to other users as well. Source_id Destination_id 20 12 29 12 34 16 The initial format looked like the above. The relationship of Follow was stored in a single table and replicated in master-slave fashion. The problem with this, however, is that throughput for writing is unsatisfactory, and indices sometimes are not loaded into the memory as the size of the table grows. A solution for this is to maintain two tables – Forward and Backward. In this solution, each table is maintained according to the directions of the source and destination, and is partitioned by user_id. Forward Table source_id destination_id updated_at X 20 12 20:50:14 X 20 13 20:51:32 20 16 Backward Table destination_id source_id updated_at X 12 20 20:50:14 X 12 32 20:51:32 12 16 Using this method enables primary key-based partitioning, while speeding up the process and making it easier to manage indices. One more important thing: for effective partitioning and replication of a table, the data should be commutative (the order of commutative and operation is interchangeable) and idempotent (the result is the same when the operation is performed several times). This precondition is made by Gizzard, Twitter's powerful sharding (or partitioning) framework. In this case an operation that has failed to be executed due to a temporary malfunction can be executed again at any time later, regardless of whether it has been executed or not. To create an internally commutative attribute, all rows have a time stamp to ensure their validity, and the updateable attribute information in a relationship becomes invalid when the time recorded in its time stamp is earlier than the time recorded in the time stamp of the row. The performance of the Social Graphs of Twitter is as follows: cardinality iteration write materialize inclusion 1ms 100 edges/ms 16 ms 1 ms Search Indices Twitter supports a search function for specific keywords, as shown above. For this, it assigns an id to every possible keyword and stores the doc_id information that includes the keyword to provide the search function. term_id doc_id 20 12 20 86 34 16 In earlier versions of Twitter, the relationship between term_id and doc_id used to be stored in a single table, as shown above. But doing this increased the size of indices, to the point that they were too big to be loaded into the RAM. Partitioning was required to solve this problem, and thus a time-based partitioning was applied, which is similar to the partitioning of tweets. Doing this has the advantage of allowing the information to be immutable, which means that the data will not be changed once it has been created. In addition, it is a quite effective partitioning as it satisfies the commutative and idempotent attributes, both of which greatly help the replication and distribution processes. Twitter plans to further improve this feature by separating partitions for document and time and replacing MySQL, the lowest-level storage, with Lucene, which is optimized for searching. Twitter Performance The following is a table showing the performance data of the current Twitter service. reads/second writes/second cardinality Tweets 100 k 850 12 b Timelines 80 k 1.2 m a lot Graphs 100 k 20 k 13 b Search 13 k 21 k 315 m The Lessons Learned Here is what we can take out of this Twitter DB decomposition by analyzing its four major types of data. There are no permanent solutions (respond flexibly to changing circumstances). There are no perfect solutions - there are only solutions that are useful for a certain amount of time. Scalability does not achieve itself - it is achieved by appropriately combining partitioning, indexing, and replication. All data related to a Read-time query must be serviced in the memory (A disk is only for permanent storage). Only a small number of problems can be solved through pre-computation. For best performance always pay attention to locality! In Conclusion We have studied Twitter's approach of throwing a stone to catch two rabbits, namely Low Latency and High Throughput. Quick readers may have already noticed that they are chasing two hares at once by creating an additional attribute that NoSQL pursues while still using a RDBMS, as the lowest-level storage. In short, Twitter assigns the commutative and idempotent attributes to data for effective partitioning and replication. From perspective, Twitter is already NoSQL-ready (It remains to be seen whether that NoSQL is Cassandra). I have learned much from Twitter while looking for the Data Store for NHN. In addition, when I was asked to write a feature article about NoSQL by The Platform Magazine, I thought it would be beneficial for people with the same concerns to share this piece of information, as doing so might reduce the time they would spend on this issue. N-Store (a tentative name), the first high-speed storage system of NHN in the future, is under development. The N-Store project members are working hard to come up with a storage system that will live up to expectations. Kim Hyeo, Development Center, NHN. If you like this article, add me on http://twitter.com/cubrid. See also http://www.slideshare.net/netik/billions-of-hits-scaling-twitter http://twitter.com/about http://engineering.twitter.com/2010/05/introducing-flockdb.html http://en.wikipedia.org/wiki/Gizzard_(scala_framework) http://gigaom.com/2010/04/07/gizzard-anyone-twitter-offers-up-code-for-distributed-data/ http://github.com/twitter/gizzard https://github.com/twitter/flockdb https://github.com/twitter/gizzmo https://github.com/robey/kestrel http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster http://www.readwriteweb.com/cloud/2011/01/how-twitter-uses-nosql.php Big Data in Real-Time at Twitter http://www.slideshare.net/nkallen/q-con-3770885 (PDF link here) [Less]

Perhaps, the HA (High-Availability) feature is one of the most important in any fully-featured database management system. It refers to an ability to minimize system down time while continuing normal operation of service in the event of hardware ... [More] , operating system, service, or network failure. This ability is a critical element in the network computing area where services should be provided 24/7. This is why it is one of the must-have Enterprise level features. The CUBRID HA feature has a shared-nothing architecture and is based on transaction log replication which guarantees 100% database consistency between master and slave database servers. This is different from that of MySQL Replication, which is "asynchronous and does not guarantee that data from the master has been replicated to the slaves" (see MySQL HA). To perform synchronous replication in MySQL, users have to either purchase external solutions or use them under open source license terms. All-in-one CUBRID Database with built-in HA feature makes it easier for users to provide continuous high-availability and 100% data consistency. There are two ways to configure the CUBRID HA to keep databases synchronized throughout multiple server systems: 1. Database Server Duplication The Figure 1 shows a common HA configuration where a database server is duplicated (indicated as a node A and node B in the figure). In case a master database server (node A) fails, the requests failover to a stand-by server (node B), which becomes a master server. Once the failed server (previously master server) is recovered, it becomes the slave database server. 2. Broker Duplication The Broker is a CUBRID-specific middleware which is a part of the CUBRID's 3-tier architecture. It relays the communication between the Database Server and external applications be it the CUBRID Manager, the JDBC or PHP Connectors. It provides functions including connection pooling, monitoring, and log tracing and analysis. The Figure 2 shows another HA configuration where a broker is duplicated (indicated as a broker b1 and broker b2 in the figure). In case a broker b1 fails, the requests failover to a broker b2. Once the failed broker (broker b1) is recovered, the requests failback to the broker b1. Such configuration is only possible in database systems with a 3-tier architecture, which CUBRID represents. CUBRID HA Development History The CUBRID HA feature was first implemented in CUBRID version 2.0 based on the Linux Heartbeat daemon which allows clients to know about the presence or disappearance of peer processes on other machines. However the Linux Heartbeat, as an external package, was not optimized for CUBRID, thus, difficult to maintain. Therefore, in CUBRID 2.2 the Development team has dropped the support for the Linux Heartbeat package and implemented the native CUBRID Heartbeat component with accent on reliability, integrity and ease of use. The CUBRID Heartbeat feature is included in the cub_master process. It exchanges heartbeat messages with cub_master processes of other nodes and executes failover on the standby server when a failure is detected. It also monitors the availability of the HA related processes (cub_server, copylogdb, applylogdb) on a regular basis. Thus, since version 2.2 the CUBRID HA feature is being developed completely in-house. These days NHN Corporation, the largest CUBRID user, deploys CUBRID with HA mode enabled in 70% of all its CUBRID deployments. HA Improvements in CUBRID 8.4.0 There are three major directions when it comes to improving the CUBRID HA feature in the 2008 R4.0 version. They are Convenience, Stability, and Functionality. Convenience As the High-Availability feature is one of the most complex implementations of any database system, DBAs report many issues pertaining to much efforts to configure, deploy, and later analyze the logs. Therefore, the CUBRID HA in version 8.4.0 is meant to bring ease of configuration, administration, and use, so that any one-month-fresh DBA can enable the HA support for their CUBRID database servers. Below is what the CUBRID HA Development Team has already performed for CUBRID 8.4.0. Revised configuration scripts that are easier to edit and reconfigure. Revised and added the CUBRID HA utilities - additional commands and tools to manage the HA feature. Revised all error codes and messages to display more comprehensible messages. Improved HA monitoring tool - the feature frequently requested by most DBAs. Improved HA logs representations to display readable outputs. Improved manual and documentations to facilitate DBAs' learning process and work. Stability For the sake of increased stability of the CUBRID HA feature, the HA Dev Team has performed intensive code refactoring, cleaning, and quality assurance. The internal algorithms are also modified and improved, especially for log duplication. For instance, see Figure 3. CUBRID HA provides three copy modes for transaction log duplication: Sync mode (most stable mode with no data loss) Async mode (fast but with data loss) Semi-sync mode (mid-fast and no data loss) In CUBRID 8.4.0 the semi-sync mode will be improved more. As a result, the CUBRID HA feature can be used in enterprise services in sync or semi-sync modes to guarantee 100% data consistency between master and slave database servers. Functionality The most important in the HA is its functionality. The HA feature in CUBRID 8.4.0 release will provide the following new features: New PHRO broker mode: PHRO (preferred host read only) is another RO mode, which enables the broker to lookup the prioritized hosts first before other hosts so that DBAs can easily control (configure) the connection order. As a result, there will be four broker modes: RW (read write), RO (read only), SO (slave only), and PHRO (preferred host read only). New M:S:R configuration: In CUBRID 8.4.0 there will be not only M:S (Master:Slave) but also M:S with multiple replicas. This configuration will provide much lower networking and disk I/O loads for heavy read loaded Web services, than the single master and multiple slaves configuration does. Multiple slaves in a single server: This is best for small size Web services whose primary goal is to lower the cost. This configuration will allow to gather all (or some) slave databases on a single host. Thus, it will be possible to allocate a single physical machine for multiple slave databases, each of which are separately replicated from their own master databases. Conclusion Currently, the CUBRID HA is known for its stability and reliability. However, the new version will be even more powerful and much more easy to use, so that even any one-month-fresh DBA can enable the HA support for their CUBRID database servers. Besides, the CUBRID HA feature will best fit the needs of enterprise services which require 100% data consistency between master and slave database servers and continuous availability."The CUBRID HA is the most distinguishing and key feature of CUBRID for Enterprises. Therefore, we will continue improving it, and provide a reliable and easy to use HA solution to ensure the data availability and consistency throughout all data servers," says Wonhee Jeon, a leader of the CUBRID HA Development Team. To learn more about other new features coming in CUBRID 8.4.0, read Roadmap: What to Expect in CUBRID 8.4.0. [Less]

Perhaps, the HA (High-Availability) feature is one of the most important in any fully-featured database management system. It refers to an ability to minimize system down time while continuing normal operation of service in the event of hardware ... [More] , operating system, service, or network failure. This ability is a critical element in the network computing area where services should be provided 24/7. This is why it is one of the must-have Enterprise level features. The CUBRID HA feature has a shared-nothing architecture and is based on transaction log replication which guarantees 100% database consistency between master and slave database servers. This is different from that of MySQL Replication, which is "asynchronous and does not guarantee that data from the master has been replicated to the slaves" (see MySQL HA). To perform synchronous replication in MySQL, users have to either purchase external solutions or use them under open source license terms. All-in-one CUBRID Database with built-in HA feature makes it easier for users to provide continuous high-availability and 100% data consistency. There are two ways to configure the CUBRID HA to keep databases synchronized throughout multiple server systems: 1. Database Server Duplication The Figure 1 shows a common HA configuration where a database server is duplicated (indicated as a node A and node B in the figure). In case a master database server (node A) fails, the requests failover to a stand-by server (node B), which becomes a master server. Once the failed server (previously master server) is recovered, it becomes the slave database server. 2. Broker Duplication The Broker is a CUBRID-specific middleware which is a part of the CUBRID's 3-tier architecture. It relays the communication between the Database Server and external applications be it the CUBRID Manager, the JDBC or PHP Connectors. It provides functions including connection pooling, monitoring, and log tracing and analysis. The Figure 2 shows another HA configuration where a broker is duplicated (indicated as a broker b1 and broker b2 in the figure). In case a broker b1 fails, the requests failover to a broker b2. Once the failed broker (broker b1) is recovered, the requests failback to the broker b1. Such configuration is only possible in database systems with a 3-tier architecture, which CUBRID represents. CUBRID HA Development History The CUBRID HA feature was first implemented in CUBRID version 2.0 based on the Linux Heartbeat daemon which allows clients to know about the presence or disappearance of peer processes on other machines. However the Linux Heartbeat, as an external package, was not optimized for CUBRID, thus, difficult to maintain. Therefore, in CUBRID 2.2 the Development team has dropped the support for the Linux Heartbeat package and implemented the native CUBRID Heartbeat component with accent on reliability, integrity and ease of use. The CUBRID Heartbeat feature is included in the cub_master process. It exchanges heartbeat messages with cub_master processes of other nodes and executes failover on the standby server when a failure is detected. It also monitors the availability of the HA related processes (cub_server, copylogdb, applylogdb) on a regular basis. Thus, since version 2.2 the CUBRID HA feature is being developed completely in-house. These days NHN Corporation, the largest CUBRID user, deploys CUBRID with HA mode enabled in 70% of all its CUBRID deployments. HA Improvements in CUBRID 8.4.0 There are three major directions when it comes to improving the CUBRID HA feature in the 2008 R4.0 version. They are Convenience, Stability, and Functionality. Convenience As the High-Availability feature is one of the most complex implementations of any database system, DBAs report many issues pertaining to much efforts to configure, deploy, and later analyze the logs. Therefore, the CUBRID HA in version 8.4.0 is meant to bring ease of configuration, administration, and use, so that any one-month-fresh DBA can enable the HA support for their CUBRID database servers. Below is what the CUBRID HA Development Team has already performed for CUBRID 8.4.0. Revised configuration scripts that are easier to edit and reconfigure. Revised and added the CUBRID HA utilities - additional commands and tools to manage the HA feature. Revised all error codes and messages to display more comprehensible messages. Improved HA monitoring tool - the feature frequently requested by most DBAs. Improved HA logs representations to display readable outputs. Improved manual and documentations to facilitate DBAs' learning process and work. Stability For the sake of increased stability of the CUBRID HA feature, the HA Dev Team has performed intensive code refactoring, cleaning, and quality assurance. The internal algorithms are also modified and improved, especially for log duplication. For instance, see Figure 3. CUBRID HA provides three copy modes for transaction log duplication: Sync mode (most stable mode with no data loss) Async mode (fast but with data loss) Semi-sync mode (mid-fast and no data loss) In CUBRID 8.4.0 the semi-sync mode will be improved more. As a result, the CUBRID HA feature can be used in enterprise services in sync or semi-sync modes to guarantee 100% data consistency between master and slave database servers. Functionality The most important in the HA is its functionality. The HA feature in CUBRID 8.4.0 release will provide the following new features: New PHRO broker mode: PHRO (preferred host read only) is another RO mode, which enables the broker to lookup the prioritized hosts first before other hosts so that DBAs can easily control (configure) the connection order. As a result, there will be four broker modes: RW (read write), RO (read only), SO (slave only), and PHRO (preferred host read only). New M:S:R configuration: In CUBRID 8.4.0 there will be not only M:S (Master:Slave) but also M:S with multiple replicas. This configuration will provide much lower networking and disk I/O loads for heavy read loaded Web services, than the single master and multiple slaves configuration does. Multiple slaves in a single server: This is best for small size Web services whose primary goal is to lower the cost. This configuration will allow to gather all (or some) slave databases on a single host. Thus, it will be possible to allocate a single physical machine for multiple slave databases, each of which are separately replicated from their own master databases. Conclusion Currently, the CUBRID HA is known for its stability and reliability. However, the new version will be even more powerful and much more easy to use, so that even any one-month-fresh DBA can enable the HA support for their CUBRID database servers. Besides, the CUBRID HA feature will best fit the needs of enterprise services which require 100% data consistency between master and slave database servers and continuous availability."The CUBRID HA is the most distinguishing and key feature of CUBRID for Enterprises. Therefore, we will continue improving it, and provide a reliable and easy to use HA solution to ensure the data availability and consistency throughout all data servers," says Wonhee Jeon, a leader of the CUBRID HA Development Team. To learn more about other new features coming in CUBRID 8.4.0, read Roadmap: What to Expect in CUBRID 8.4.0. [Less]

Below is a brief comparison of the key feature development speed between CUBRID and MySQL.