Summary

Allow users to store different data sets in different namespaces that are all managed by the same Concourse server.

Motivation

This feature will allow us to build a demo web app that allows people to try Concourse via CaSH because we can store the data for each user session in a different namespace. Additionally, this feature will provide additional parity with other databases that allow users to separate different datasets within the same server instance.

Use Cases

• I do not care about namespaces and just want to use Concourse
• I want to easily separate data sets into different namespaces so that my queries, etc don't read unnecessary data
• I want to easily delete namespaces that contain data I don't care about

A Word of Caution

Concourse is schemaless and seeks to impose little requirements on the ways in which users model data. Therefore, we need to be sure that this feature does not violate that intent or any of the Design Principles. This feature is meant to give users greater flexibility in how they use Concourse, but must not impose any additional mandatory configuration compared to previous versions. 

Important Questions and Semantics

1. Each namespace should be associated with its own Engine
2. There is a "default" namespace that is always started when the server starts
3. The server should hold a Map<String, Engine> that contains a mapping from namespace to the corresponding Engine
4. Each method in the thrift api must be updated to accept a "namespace" parameter. 
   1. If the param isn't provided, then the default namespace is used
5. The namespace for each client should be supplied upon connection.
   1. This means that the client cannot change its namespace
   2. If no namespace is provided when connecting, then use the default namespace
6. Data is stored in the following formats:
   1. ${database_directory}/namespace/
   2. ${buffer_directory}/namespace
   3. This means we need an upgrade task that will migrate all existing data to the default namespace
7. Data is unique and cannot be shared across namespaces
8. Namespaces are created dynamically if they do not exist
9. A user has permissions for all namespaces
   1. In a future release, we can probably limit permissions by namespaces
10. The dump tool CLI must take an optional namespace argument
11. How are server level transactions impacted by the fact that each server level method must have a namespace specified? What if the client is in Transaction, how does the transaction know which namespace/engine to commit? How do we make sure it does not get confused?

Effects on Public API

The following methods will be added to the Public API and will therefore have a client-side implementation:

• public static Concourse connect(String host, int port, String username, String password, String namespace)
• Edit #connect() to account for a prefs file that both does and does not have the namespace specified

Effects on Thrift API

• Each method in the Thrift API will need to take a namespace parameter

Effects on JMX API

• Add a new #deleteNamespace() managed operation

Implementation Plan

Task	Description	Notes
Remove static state from LockService	Remove all static state and fix the Engine such that it takes a LockService object upon instantiation. An AtomicOperation will also need to take a LockService object, which will be passed from the Engine or Transaction that is its parent.
Remove static state from RangeLockService	Remove all static state and fix the Engine such that it takes a RangeLockService object upon instantiation. An AtomicOperation will also need to take a RangeLockService object, which will be passed from the Engine or Transaction that is its parent.
Edit Thrift API	Each method of the Thrift API should take a "namespace" parameter
Add support for multiple Engines in the server	ConcourseServer should maintain a mapping from namespace to Engine. For each method in the server, it should use the Engine that corresponds to the passed in namespace. There should be a default namespace Engine that is always loaded on server startup. On server shutdown, each engine must be stopped. A namespace should be created on the fly if it dos not exist when the user connects and attempts to use that namespace.
Change storage paths to account for namespace	Make sure that each Engine uses ${database_directory}/namespace and ${buffer_directory}/namespace as the bases for file storage
Edit Public API	Edit the public api to add a connection method with a namespace parameter
Edit concourse_client.prefs	Add optional "namespace" key to concourse_client.prefs.
Edit dump tool CLI	Add an optional namespace parameter for the dump tool cli
Upgrade Task	Add an upgrade task for 0.3 versions to have their data moved to the correct directory to account for it being in the default namespace
Delete namespace CLI	Create a CLI to delete a namespace