Summary

Tackle any obvious low hanging fruit that will incrementally improve write/read performance that users experience.

Motivation

Early write performance seems somewhat slow compared to other database systems. Read performance seems okay for client and servers on the same box. For the system to scale and be usable in a distributed manner, we really need to make performance better. This project will look into low hanging JVM/network/etc tuning to see if we can improve performance incrementally. If possible, we may also look into architecture level improvements.

Use Cases

• I want to write data to Concourse at least as fast as I can with MySQL with comparable conditions
• I want to read/write data to Concourse over the network and I don't want it to take a long time

Potential Improvements

This is a running list of broad areas that may provide incremental performance boosts.

• New JVM Args
  • -XX:+UseThreadPriorities
  • -XX:ThreadPriorityPolicy=42
    • allows lowering thread priority without being root. see http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workaround.html
  • -XX:+HeapDumpOnOutOfMemoryError
  • See https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh for more ideas
• Thread Prioritization
  • http://www.javamex.com/tutorials/threads/priority_what.shtml
• Use TCompactProtocol as opposed to the TBinaryProtocol
• Use Trove
• http://www.slideshare.net/aszegedi/everything-i-ever-learned-about-jvm-performance-tuning-twitter
• http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
• http://www.oracle.com/technetwork/java/tuning-139912.html#section4.2.5

Bugs