This is high level overview of the codebase meant to give devs some direction before diving in. There is a lot of nuance that is not covered here (i.e. concurrency controls, atomic operations, managed operations, serialization, etc), but hopefully this is helpful in orienting how people look at the codebase and provides a spring board for further questions. If you do have follow up questions, please send them to the concourse-devs@googlegroups.com list so everyone can benefit!
Overview
The concourse project contains the client code and the concourse-server project contains the server code. The client and server talk to each other using thrift. The concourse-shell project contains the code for an interactive command line client that can be used to interact with a server.
concourse
The client code consists of some shared utility classes, but the most interesting stuff is in the Concourse.java file, which contains both the public API definition and implementation (Concourse.Client). The client code doesn't do a ton of work. In most cases, it just
- takes input from the user
- converts the input to a form that can be serialized by Thrift (i.e. every java Object gets converted to a TObject)
- sends the transformed data over the wire using Thrift
- waits for the server to send a response back
- transforms the server's response to a more user friendly format (i.e. every TObject gets converted to a java Object, some java collections get converted to formats with better toString output, etc).
We only really need to change client code if we're adding stuff to the public api because the client code should not really contain a lot of core logic.
concourse-server
The server code is where most of the action happens. The most important concept is the Engine. In general, the workflow is that the user passes data/requests to Concourse. Concourse then sends the data/request over the wire to ConcourseServer. Finally, ConcourseServer passes the data/request to the Engine, which actually does the work to store/retrieve data.
While the Engine is the most interesting concept, it actually does very little work. For the most part, it delegates requests to store or retrieve data to its subcomponents: the Buffer and the Database. Concourse's storage model is a system that initially stores data in a temporary buffer before transporting and indexing the data in a permanent database. The Buffer and Database classes contain the logic for handling data in their respective formats and the Engine class contains the logic for reconciling (possibly conflicting) results from the two stores and ensuring that data is passed between them in a safe and consistent manner.
The Buffer is fairly straight forward linked list of revisions. The Database, on the other hand, is a complex system that stores three different views of data in Blocks on disk and cached Records in memory. The Block classes contain logic for efficiently sorting data on the fly, writing data to disk and reading data from disk into a Record in memory. The Record classes contain logic for reading data for a specific query.
Similar to the Engine, you have a Transaction. A Transaction is another mini Engine that is used to handle cases where the client wants to stage data/requests in an ACID transaction. Just like the Engine, a Transaction is a BufferedStore which initially stores data in a in-memory Queue and eventually transports the data to the Engine when the user commits the transaction.
The role of the ConcourseServer class is to take requests from the client and to decide if they should be sent directly to the Engine or if it should be handled in a Transaction first. Additionally, the ConcourseServer class manages security using anAccessManager.
concourse-shell
This shell client (CaSH) is a simple interpreter that allows a user to interact with Concourse using the public api with dynamic groovy code. The main class is ConcourseShell.java. This class continuously takes input from the user and passes it to a client connection.
Hopefully this overview is somewhat helpful. The spec pages listed on the roadmap contain descriptions about how to take high level features and approach their implementation in the codebase as well. Definitely reach out if you have further questions. I'm happy to help.