Getting Started
Introduction
This document is meant to offer first time users a few easy steps to install the latest version of Concourse Server and read/write data using the Concourse Action SHell (CaSH). Concourse is very simple to use and works right out the box–no assembly required. After completing these steps, you should have a fully functional Concourse deployment, but we'll also discuss a few configuration parameters you may want to tweak before using your Concourse installation in production.
Install Concourse
In this section you will download the latest version of the concourse-server
package and set it up on your system.
Download the latest release
From the terminal, navigate to the location where you wish to install Concourse Server and download the latest release
$ curl -O -L -J http://cinchapi.org/concourse/download
Run the installer package
Once the download finishes you will have a .bin
file that contains the Concourse Server distribution. Execute this installer to verify the integrity of your download and create a concourse-server
directory that contains all the Concourse Server files.
$ sh concourse-server-0.3.0.1179.bin Creating directory concourse-server Verifying archive integrity... All good. Uncompressing Concourse Server..................................................................... $ cd concourse-server
Sanity check the installation
Make sure that your system is properly configured by starting Concourse server in console
mode to make sure that no errors occur.
$ ./bin/concourse console Running Concourse Server... wrapper | --> Wrapper Started as Console wrapper | Java Service Wrapper Community Edition 64-bit 3.5.22 wrapper | Copyright (C) 1999-2013 Tanuki Software, Ltd. All Rights Reserved. wrapper | http://wrapper.tanukisoftware.com wrapper | wrapper | Launching a JVM... jvm 1 | WrapperManager: Initializing... jvm 1 | _____ jvm 1 | / __ \ jvm 1 | | / \/ ___ _ __ ___ ___ _ _ _ __ ___ ___ jvm 1 | | | / _ \| '_ \ / __/ _ \| | | | '__/ __|/ _ \ jvm 1 | | \__/\ (_) | | | | (_| (_) | |_| | | \__ \ __/ jvm 1 | \____/\___/|_| |_|\___\___/ \__,_|_| |___/\___| jvm 1 | jvm 1 | Copyright (c) 2013, Cinchapi Software Collective, LLC. All Rights Reserved. jvm 1 | The Concourse server has started
If all goes well, you should see output similar to that above. Any errors that prevent the server from starting will be displayed in the console with more information possibly located in log/error.log
.
Start the server
Press ctrl + c
to stop the server. Now start Concourse again in background
mode as a daemon process.
$ ./bin/concourse start Starting Concourse Server... Waiting for Concourse Server... . running: PID:23969
Congratulations! You now have Concourse Server up and running. You can now read and write data using CaSH or the Concourse API.
Pro Tip
You can also use the bin/start
and bin/stop
scripts to manage the Concourse Server lifecycle.
Explore CaSH
In this section you will launch the Concourse Action SHell (CaSH) client and get familiar with its mode of operation.
Launch CaSH
CaSH is a command line interpreter that executes commands against Concourse Server using the standard Java API. CaSH is backed by the full power of the Groovy scripting language so you can write powerful routines that read and wrie data on the fly.
Enter the default admin password of admin
when prompted.
$ ./bin/cash Password: ***** _____ / __ \ | / \/ ___ _ __ ___ ___ _ _ _ __ ___ ___ | | / _ \| '_ \ / __/ _ \| | | | '__/ __|/ _ \ | \__/\ (_) | | | | (_| (_) | |_| | | \__ \ __/ \____/\___/|_| |_|\___\___/ \__,_|_| |___/\___| Copyright (c) 2013, Cinchapi Software Collective, LLC. All Rights Reserved. Client Version 0.3.0.193 Server Version 0.3.0.193 Type HELP for help. Type EXIT to quit. cash$
Documentation
You can display the help for CaSH and see documentation about the shell and the Concourse API at any time by typing help
or man
.
Pro Tip
Press q
to exit the HELP page and return to CaSH.
CASH(1) CASH(1) NAME cash - Concourse Action SHell DESCRIPTION Cash is an interpreter that executes commands against Concourse using the standard API. Since Cash is backed by the full power of the Groovy scripting language, it is possible to write powerful routines that read and write data on the fly. USAGE command [arg1][, arg2][...][, argN] COMMANDS concourse.add key, value, record -> boolean Add key (string) AS value (object) TO record (long) if does not exist. concourse.audit record -> Map<Timestamp,String> Return a log of every revision for record (long). concourse.audit key, record -> Map<Timestamp,String> Return a log of every revision for key (string) IN record (long). concourse.clear key, record Remove every mapping from key (string) in record (long). concourse.create -> long Create a new record and return its primary key. :
Tab Completion
CaSH is equipped with tab complete so you have a handy reference when you need to know which API commands are available. Just start typing and press tab
.
cash$ c concourse.add concourse.audit concourse.clear concourse.create concourse.describe concourse.fetch concourse.find concourse.get concourse.link concourse.ping concourse.remove concourse.revert concourse.search concourse.set concourse.unlink concourse.verify cash$ concourse.
Calling API Functions
All API functions are prefixed with concourse.
You can call a function using standard java syntax...
cash$ concourse.add(key, value, record)
or groovy shortcut syntax (no parenthesis required).
cash$ concourse.add key, value, record
Pro Tip
You can cycle through your previous commands using the up
and down
arrow keys.
Timestamps
CaSH provides a time()
function to convert a Unix timestamp (in microseconds) or an English phrase to a Timestamp
object, which is necessary to perform historical operations.
Convert a Unix timestamp
cash$ time(1387899621424000) Returned '2013-12-24T09:40:21.424-06:00' in 15 ms
Convert an English phrase
cash$ time("today") Returned '2013-12-24T00:00:00.000-06:00' in 17 ms cash$ time("last week") Returned '2013-12-17T09:51:53.793-06:00' in 15 ms cash$ time("yesterday") Returned '2013-12-23T09:51:27.863-06:00' in 14 ms cash$ time("April 1, 1997") Returned '1997-04-01T00:00:00.000-06:00' in 13 ms
Pro Tip
The HELP
page describes all the possible inputs for the time()
function in the DATETIME
section.
Pro Tip
The date()
function is an alias for time()
.
Query Operators
CaSH provides shortcuts for the query Operators
used in the find()
functions.
CaSH shortcut | Operator |
---|---|
eq | EQUALS |
ne | NOT_EQUALS |
gt | GREATER_THAN |
gte | GREATER_THAN_OR_EQUALS |
lt | LESS_THAN |
lte | LESS_THAN_OR_EQUALS |
bw | BETWEEN |
regex | REGEX |
nregex | NOT_REGEX |
Use Concourse
In this section you will read and write data using some of the operations available in the API. Since Concourse is schemaless, you don't have to create any structure (i.e. keyspaces, tables, etc) before reading or writing data. All keys and records are created on the fly.
The Basics
Applications need to efficiently read and write data and Concourse makes it very easy to add, remove, fetch and query data.
Add
The add
function appends a value to a key in a record. A record many contain multiple values and those values can be of various types.
Add name
as John Doe
to record 1
.
cash$ concourse.add "name", "John Doe", 1 Returned 'true' in 92 ms
Also add name
as Johhny Doe
, Jonathan Doe
and J. Doe
to record 1
.
cash$ concourse.add "name", "Johnny Doe", 1 Returned 'true' in 28 ms cash$ concourse.add "name", "Jonathan Doe", 1 Returned 'true' in 24 ms cash$ concourse.add "name", "J. Doe", 1 Returned 'true' in 24 ms
Add age
as 30
, 30.5
, '30'
and true
to record 1
.
cash$ concourse.add "age", 30, 1 Returned 'true' in 28 ms cash$ concourse.add "age", 30.5F, 1 Returned 'true' in 26 ms cash$ concourse.add "age", "30", 1 Returned 'true' in 28 ms cash$ concourse.add "age", true, 1 Returned 'true' in 25 ms
Remove
The remove
function deletes a previously added value from a key in a record.
Remove age
as true
from record 1
.
cash$ concourse.remove "age", true, 1
Fetch
The fetch
function retrieves all the values that exist for a key in a record.
Fetch name
from record 1
.
cash$ concourse.fetch "name", 1 Returned '[John Doe, Johnny Doe, Jonathan Doe, J. Doe]' in 23 ms
Get
The get
function retrieves the oldest value that exists for a key in a record.
Get name
from record 1
.
cash$ concourse.get "name", 1 Returned 'John Doe' in 27 ms
Set
The set
function atomically removes all previously added data from a key in a record before adding the specified value.
Use a for loop to add multiple values to baz
in record 1
and then set baz
as 6 in record 1.
cash$ for(int i = 0; i < 5; i++){concourse.add "baz", i, 1} Completed in 43 ms cash$ concourse.fetch "baz", 1 Returned '[0, 1, 2, 3, 4]' in 50 ms cash$ concourse.set "baz", 6, 1 Completed in 54 ms cash$ concourse.fetch "baz", 1 Returned '[6]' in 21 ms
Describe
The describe
function lists all the non-empty keys in a record.
Describe record 1
.
cash$ concourse.describe 1 Returned '[name, age, baz]' in 23 ms
Verify
The verify
function returns true if the specified value exists for the key in the record.
Verify age
as 30
in record 1
.
cash$ concourse.verify "age", 30, 1 Returned 'true' in 28 ms
Find
The find
function queries the database for records that match a criteria that is defined by a key which has at least one value that satisfies a operator in relation to the specified values. Concourse indexes data on the fly, so you can use this function on any key without explicitly creating a secondary index.
Pro Tip
The find()
function is simlar to the SELECT statement in SQL.
Create a for loop from 1 to 1000 and add count as i
in record i
and then find all the records with a value for count
between 100
and 300
.
The BETWEEN operator is inclusive of the smaller value and exclusive of the larger one.
cash$ for(int i = 1; i <= 1000; i++){ concourse.add "count", i, i} Completed in 866 ms cash$ concourse.find "count", bw, 100, 300 Returned '[100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299]' in 199 ms
Version Control
Concourse is a version control database and provides several functions to both track and manage changes to data.
Audit
The audit
function lists all the changes to a record or a key in a record.
Audit record 1.
cash$ concourse.audit 1 Returned ' +--------------------------------------------------------------------------------------------+ | DateTime | Revision | +--------------------------------------------------------------------------------------------+ | 2013-12-26T10:00:08.624-06:00 | ADD name AS John Doe (STRING) IN 1 AT 1388073608624000 | | 2013-12-26T10:00:17.954-06:00 | ADD name AS Johnny Doe (STRING) IN 1 AT 1388073617954000 | | 2013-12-26T10:00:23.859-06:00 | ADD name AS Jonathan Doe (STRING) IN 1 AT 1388073623859000 | | 2013-12-26T10:00:28.412-06:00 | ADD name AS J. Doe (STRING) IN 1 AT 1388073628412000 | | 2013-12-26T10:00:38.249-06:00 | ADD age AS 30 (INTEGER) IN 1 AT 1388073638249000 | | 2013-12-26T10:00:43.338-06:00 | ADD age AS 30.5 (FLOAT) IN 1 AT 1388073643338000 | | 2013-12-26T10:00:49.834-06:00 | ADD age AS 30 (STRING) IN 1 AT 1388073649834000 | | 2013-12-26T10:00:54.599-06:00 | ADD age AS true (BOOLEAN) IN 1 AT 1388073654599000 | | 2013-12-26T10:01:00.678-06:00 | REMOVE age AS true (BOOLEAN) IN 1 AT 1388073660678000 | | 2013-12-26T10:01:38.086-06:00 | ADD baz AS 0 (INTEGER) IN 1 AT 1388073698086000 | | 2013-12-26T10:01:38.089-06:00 | ADD baz AS 1 (INTEGER) IN 1 AT 1388073698089000 | | 2013-12-26T10:01:38.090-06:00 | ADD baz AS 2 (INTEGER) IN 1 AT 1388073698090000 | | 2013-12-26T10:01:38.092-06:00 | ADD baz AS 3 (INTEGER) IN 1 AT 1388073698092000 | | 2013-12-26T10:01:38.093-06:00 | ADD baz AS 4 (INTEGER) IN 1 AT 1388073698093000 | | 2013-12-26T10:01:50.478-06:00 | REMOVE baz AS 0 (INTEGER) IN 1 AT 1388073710478000 | | 2013-12-26T10:01:50.481-06:00 | REMOVE baz AS 1 (INTEGER) IN 1 AT 1388073710481000 | | 2013-12-26T10:01:50.482-06:00 | REMOVE baz AS 2 (INTEGER) IN 1 AT 1388073710482000 | | 2013-12-26T10:01:50.483-06:00 | REMOVE baz AS 3 (INTEGER) IN 1 AT 1388073710483000 | | 2013-12-26T10:01:50.484-06:00 | REMOVE baz AS 4 (INTEGER) IN 1 AT 1388073710484000 | | 2013-12-26T10:01:50.486-06:00 | ADD baz AS 6 (INTEGER) IN 1 AT 1388073710486000 | | 2013-12-26T10:03:46.305-06:00 | ADD count AS 1 (INTEGER) IN 1 AT 1388073826305000 | +--------------------------------------------------------------------------------------------+ ' in 17 ms
Audit age
in record 1
.
cash$ concourse.audit "age", 1 Returned ' +---------------------------------------------------------------------------------------+ | DateTime | Revision | +---------------------------------------------------------------------------------------+ | 2013-12-24T09:40:21.424-06:00 | ADD age AS 30 (INTEGER) IN 1 AT 1387899621424000 | | 2013-12-24T09:40:31.651-06:00 | ADD age AS 30.5 (STRING) IN 1 AT 1387899631651000 | | 2013-12-24T09:40:37.574-06:00 | ADD age AS 30 (STRING) IN 1 AT 1387899637574000 | | 2013-12-24T09:40:41.976-06:00 | ADD age AS true (BOOLEAN) IN 1 AT 1387899641976000 | | 2013-12-24T09:40:49.429-06:00 | REMOVE age AS true (BOOLEAN) IN 1 AT 1387899649429000 | +---------------------------------------------------------------------------------------+ ' in 17 ms
Revert
The revert
function atomically returns a key in a record to a previous state (specified by a Timestamp
).
Revert baz
in record 1
to the timestamp when 2 was added.
cash$ it = concourse.audit(1).keySet().iterator(); for(int i = 0; i < 11; i++){it.next(); }; t1 = it.next(); // save the relevant timestamp to t1 cash$ concourse.revert "baz", 1, t1 Completed in 46 ms concourse.fetch "baz", 1 Returned '[0, 1, 2]' in 16 ms
Historical Operations
Concourse allows you to describe
, fetch
, verify
, and find
data from the past.
Describe record 1
yesterday.
cash$ concourse.describe 1, time("yesterday") Returned '[]' in 23 ms
Describe record 1
at the timestamp before the first value for baz
was added.
cash$ it = concourse.audit(1).keySet().iterator(); for(int i = 0; i < 8; i++){it.next(); }; t1 = it.next(); // save the relevant timestamp to t1 cash$ concourse.describe 1, time((t1.getMicros() - 1)) Returned '[name, age]' in 25 ms
Fetch name
from record 1
at the timestamp when you added name
as Johnny Doe.
cash$ it = concourse.audit(1).keySet().iterator(); for(int i = 0; i < 1; i++){it.next(); }; t1 = it.next(); // save the relevant timestamp to t1 cash$ concourse.fetch "name", 1, t1 Returned '[John Doe, Johnny Doe]' in 20 ms
Find count
greater than 50
at the timestamp when count
as 50
was added
cash$ t1 = concourse.audit(50).keySet().iterator().next(); // save the relevant timestamp to t1 cash$ concourse.find "count", gt, 50, t1 Returned '[]' in 106 ms
Find count
greater than 50
at the timestamp when count
as 500
was added.
cash$ t1 = concourse.audit(500).keySet().iterator().next(); // save the relevant timestamp to t1 cash$ concourse.find "count", gt, 50, t1 Returned '[51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500]' in 252 ms
Verify age
as true
in record 1
at the timestamp when you added age
as true
in record 1
.
cash$ it = concourse.audit("age", 1).keySet().iterator(); for(int i = 0; i < 3; i++){it.next(); }; t1 = it.next(); // save the relevant timestamp to t1 cash$ concourse.verify "age", true, 1, t1 Returned 'true' in 14 ms
Searching
TODO: finishme
Concourse automatically indexes text data for full text substring search
cash$ strings = ["The Cat in the Hat", "Green Eggs and Ham", "Horton Hears a Who", "The Cat in the Hat Comes Back", "Scrambled Eggs Super"] Returned '[The Cat in the Hat, Green Eggs and Ham, Horton Hears a Who, The Cat in the Hat Comes Back, Scrambled Eggs Super]' in 340 ms cash$ for(int i = 0; i < 1000; i++){ value = strings[i % strings.size()]; concourse.add "title", value, i} Completed in 487 ms cash$ concourse.search "title", "eggs" Returned '[956, 959, 954, 889, 949, 891, 951, 944, 946, 894, 941, 936, 939, 994, 934, 929, 996, 999, 931, 986, 926, 984, 924, 991, 989, 921, 979, 919, 976, 916, 914, 981, 971, 911, 969, 909, 974, 906, 904, 961, 901, 899, 966, 964, 896, 1, 4, 6, 9, 11, 14, 16, 19, 21, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 111, 114, 116, 119, 121, 124, 126, 129, 131, 134, 136, 139, 141, 144, 146, 149, 151, 154, 156, 159, 161, 164, 166, 169, 171, 174, 176, 179, 181, 184, 186, 189, 191, 194, 196, 199, 201, 204, 206, 209, 211, 214, 216, 219, 221, 224, 226, 229, 231, 234, 236, 239, 241, 244, 246, 249, 251, 254, 256, 259, 261, 264, 266, 269, 271, 274, 276, 279, 281, 284, 286, 289, 291, 294, 296, 299, 301, 304, 306, 309, 311, 314, 316, 319, 321, 324, 326, 329, 331, 334, 336, 339, 341, 344, 346, 349, 351, 354, 356, 359, 361, 364, 366, 369, 371, 374, 376, 379, 381, 384, 386, 389, 391, 394, 396, 399, 401, 404, 406, 409, 411, 414, 416, 419, 421, 424, 426, 429, 431, 434, 436, 439, 441, 444, 446, 449, 451, 454, 456, 459, 461, 464, 466, 469, 471, 474, 476, 479, 481, 484, 486, 489, 491, 494, 496, 499, 501, 504, 506, 509, 511, 514, 516, 519, 521, 524, 526, 529, 531, 534, 536, 539, 541, 544, 546, 549, 551, 554, 556, 559, 561, 564, 566, 569, 571, 574, 576, 579, 581, 584, 586, 589, 591, 594, 596, 599, 601, 604, 606, 609, 611, 614, 616, 619, 621, 624, 626, 629, 631, 634, 636, 639, 641, 644, 646, 649, 651, 654, 656, 659, 661, 664, 666, 669, 671, 674, 676, 679, 681, 684, 686, 689, 691, 694, 696, 699, 701, 704, 706, 709, 711, 714, 716, 719, 721, 724, 726, 729, 731, 734, 736, 739, 741, 744, 746, 749, 751, 754, 756, 759, 761, 764, 766, 769, 771, 774, 776, 779, 781, 784, 786, 789, 791, 794, 796, 799, 801, 804, 806, 809, 811, 814, 816, 819, 821, 824, 826, 829, 831, 834, 836, 839, 841, 844, 846, 849, 851, 854, 856, 859, 861, 864, 866, 869, 871, 874, 876, 879, 881, 884, 886]' in 86 ms
Next Steps
Try the TwitterCLI Exercise.