Getting Started
- Jeff Nelson
Introduction
This document is meant to offer first time users a few easy steps to install the latest version of Concourse Server and read/write data using the Concourse Action SHell (CaSH). Concourse is very simple to use and works right out the box–no assembly required. After completing these steps, you should have a fully functional Concourse deployment, but we'll also discuss a few configuration parameters you may want to tweak before using your Concourse installation in production.
Install Concourse
In this section you will download the latest version of the concourse-server package and set it up on your system.
Download the latest release
From the terminal, navigate to the location where you wish to install Concourse Server and download the latest release
$ curl -O -L -J http://cinchapi.org/concourse/download
Run the installer package
Once the download finishes you will have a .bin file that contains the Concourse Server distribution. Execute this installer to verify the integrity of your download and create a concourse-server directory that contains all the Concourse Server files.
$ sh concourse-server-0.3.0.1179.bin Creating directory concourse-server Verifying archive integrity... All good. Uncompressing Concourse Server..................................................................... $ cd concourse-server
Sanity check the installation
Make sure that your system is properly configured by starting Concourse server in console mode to make sure that no errors occur.
$ ./bin/concourse console Running Concourse Server... wrapper | --> Wrapper Started as Console wrapper | Java Service Wrapper Community Edition 64-bit 3.5.22 wrapper | Copyright (C) 1999-2013 Tanuki Software, Ltd. All Rights Reserved. wrapper | http://wrapper.tanukisoftware.com wrapper | wrapper | Launching a JVM... jvm 1 | WrapperManager: Initializing... jvm 1 | _____ jvm 1 | / __ \ jvm 1 | | / \/ ___ _ __ ___ ___ _ _ _ __ ___ ___ jvm 1 | | | / _ \| '_ \ / __/ _ \| | | | '__/ __|/ _ \ jvm 1 | | \__/\ (_) | | | | (_| (_) | |_| | | \__ \ __/ jvm 1 | \____/\___/|_| |_|\___\___/ \__,_|_| |___/\___| jvm 1 | jvm 1 | Copyright (c) 2013, Cinchapi Software Collective, LLC. All Rights Reserved. jvm 1 | The Concourse server has started
If all goes well, you should see output similar to that above. Any errors that prevent the server from starting will be displayed in the console with more information possibly located in log/error.log.
Start the server
Press ctrl + c to stop the server. Now start Concourse again in background mode as a daemon process.
$ ./bin/concourse start Starting Concourse Server... Waiting for Concourse Server... . running: PID:23969
Congratulations! You now have Concourse Server up and running. You can now read and write data using CaSH or the Concourse API.
Pro Tip
You can also use the bin/start and bin/stop scripts to manage the Concourse Server lifecycle.
Explore CaSH
In this section you will launch the Concourse Action SHell (CaSH) client and get familiar with its mode of operation.
Launch CaSH
CaSH is a command line interpreter that executes commands against Concourse Server using the standard Java API. CaSH is backed by the full power of the Groovy scripting language so you can write powerful routines that read and wrie data on the fly.
Enter the default admin password of admin when prompted.
$ ./bin/cash Password: ***** _____ / __ \ | / \/ ___ _ __ ___ ___ _ _ _ __ ___ ___ | | / _ \| '_ \ / __/ _ \| | | | '__/ __|/ _ \ | \__/\ (_) | | | | (_| (_) | |_| | | \__ \ __/ \____/\___/|_| |_|\___\___/ \__,_|_| |___/\___| Copyright (c) 2013, Cinchapi Software Collective, LLC. All Rights Reserved. Client Version 0.3.0.193 Server Version 0.3.0.193 Type HELP for help. Type EXIT to quit. cash$
Documentation
You can display the help for CaSH and see documentation about the shell and the Concourse API at any time by typing help or man.
Pro Tip
Press q to exit the HELP page and return to CaSH.
CASH(1) CASH(1)
NAME
cash - Concourse Action SHell
DESCRIPTION
Cash is an interpreter that executes commands against Concourse using the standard API.
Since Cash is backed by the full power of the Groovy scripting language, it is possible
to write powerful routines that read and write data on the fly.
USAGE
command [arg1][, arg2][...][, argN]
COMMANDS
concourse.add key, value, record -> boolean
Add key (string) AS value (object) TO record (long) if does not exist.
concourse.audit record -> Map<Timestamp,String>
Return a log of every revision for record (long).
concourse.audit key, record -> Map<Timestamp,String>
Return a log of every revision for key (string) IN record (long).
concourse.clear key, record
Remove every mapping from key (string) in record (long).
concourse.create -> long
Create a new record and return its primary key.
:
Tab Completion
CaSH is equipped with tab complete so you have a handy reference when you need to know which API commands are available. Just start typing and press tab.
cash$ c concourse.add concourse.audit concourse.clear concourse.create concourse.describe concourse.fetch concourse.find concourse.get concourse.link concourse.ping concourse.remove concourse.revert concourse.search concourse.set concourse.unlink concourse.verify cash$ concourse.
Calling API Functions
All API functions are prefixed with concourse. You can call a function using standard java syntax...
cash$ concourse.add(key, value, record)
or groovy shortcut syntax (no parenthesis required).
cash$ concourse.add key, value, record
Pro Tip
You can cycle through your previous commands using the up and down arrow keys.
Timestamps
CaSH provides a time() function to convert a Unix timestamp (in microseconds) or an English phrase to a Timestamp object, which is necessary to perform historical operations.
Convert a Unix timestamp
cash$ time(1387899621424000) Returned '2013-12-24T09:40:21.424-06:00' in 15 ms
Convert an English phrase
cash$ time("today")
Returned '2013-12-24T00:00:00.000-06:00' in 17 ms
cash$ time("last week")
Returned '2013-12-17T09:51:53.793-06:00' in 15 ms
cash$ time("yesterday")
Returned '2013-12-23T09:51:27.863-06:00' in 14 ms
cash$ time("April 1, 1997")
Returned '1997-04-01T00:00:00.000-06:00' in 13 ms
Pro Tip
The HELP page describes all the possible inputs for the time() function in the DATETIME section.
Pro Tip
The date() function is an alias for time().
Query Operators
CaSH provides shortcuts for the query Operators used in the find() functions.
| CaSH shortcut | Operator |
|---|---|
eq | EQUALS |
ne | NOT_EQUALS |
gt | GREATER_THAN |
gte | GREATER_THAN_OR_EQUALS |
lt | LESS_THAN |
lte | LESS_THAN_OR_EQUALS |
bw | BETWEEN |
regex | REGEX |
nregex | NOT_REGEX |
Use Concourse
In this section you will read and write data using some of the operations available in the API. Since Concourse is schemaless, you don't have to create any structure (i.e. keyspaces, tables, etc) before reading or writing data. All keys and records are created on the fly.
The Basics
Applications need to efficiently read and write data and Concourse makes it very easy to add, remove, fetch and query data.
Add
The add function appends a value to a key in a record. A record many contain multiple values and those values can be of various types.
Add name as John Doe to record 1.
cash$ concourse.add "name", "John Doe", 1 Returned 'true' in 92 ms
Also add name as Johhny Doe, Jonathan Doe and J. Doe to record 1.
cash$ concourse.add "name", "Johnny Doe", 1 Returned 'true' in 28 ms cash$ concourse.add "name", "Jonathan Doe", 1 Returned 'true' in 24 ms cash$ concourse.add "name", "J. Doe", 1 Returned 'true' in 24 ms
Add age as 30, 30.5, '30' and true to record 1.
cash$ concourse.add "age", 30, 1 Returned 'true' in 28 ms cash$ concourse.add "age", 30.5F, 1 Returned 'true' in 26 ms cash$ concourse.add "age", "30", 1 Returned 'true' in 28 ms cash$ concourse.add "age", true, 1 Returned 'true' in 25 ms
Remove
The remove function deletes a previously added value from a key in a record.
Remove age as true from record 1.
cash$ concourse.remove "age", true, 1
Fetch
The fetch function retrieves all the values that exist for a key in a record.
Fetch name from record 1.
cash$ concourse.fetch "name", 1 Returned '[John Doe, Johnny Doe, Jonathan Doe, J. Doe]' in 23 ms
Get
The get function retrieves the oldest value that exists for a key in a record.
Get name from record 1.
cash$ concourse.get "name", 1 Returned 'John Doe' in 27 ms
Set
The set function atomically removes all previously added data from a key in a record before adding the specified value.
Use a for loop to add multiple values to baz in record 1 and then set baz as 6 in record 1.
cash$ for(int i = 0; i < 5; i++){concourse.add "baz", i, 1}
Completed in 43 ms
cash$ concourse.fetch "baz", 1
Returned '[0, 1, 2, 3, 4]' in 50 ms
cash$ concourse.set "baz", 6, 1
Completed in 54 ms
cash$ concourse.fetch "baz", 1
Returned '[6]' in 21 ms
Describe
The describe function lists all the non-empty keys in a record.
Describe record 1.
cash$ concourse.describe 1 Returned '[name, age, baz]' in 23 ms
Verify
The verify function returns true if the specified value exists for the key in the record.
Verify age as 30 in record 1.
cash$ concourse.verify "age", 30, 1 Returned 'true' in 28 ms
Find
The find function queries the database for records that match a criteria that is defined by a key which has at least one value that satisfies a operator in relation to the specified values. Concourse indexes data on the fly, so you can use this function on any key without explicitly creating a secondary index.
Pro Tip
The find() function is simlar to the SELECT statement in SQL.
Create a for loop from 1 to 1000 and add count as i in record i and then find all the records with a value for count between 100 and 300.
The BETWEEN operator is inclusive of the smaller value and exclusive of the larger one.
cash$ for(int i = 1; i <= 1000; i++){ concourse.add "count", i, i}
Completed in 866 ms
cash$ concourse.find "count", bw, 100, 300
Returned '[100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299]' in 199 ms
Version Control
Concourse is a version control database and provides several functions to both track and manage changes to data.
Audit
The audit function lists all the changes to a record or a key in a record.
Audit record 1.
cash$ concourse.audit 1 Returned ' +--------------------------------------------------------------------------------------------+ | DateTime | Revision | +--------------------------------------------------------------------------------------------+ | 2013-12-26T10:00:08.624-06:00 | ADD name AS John Doe (STRING) IN 1 AT 1388073608624000 | | 2013-12-26T10:00:17.954-06:00 | ADD name AS Johnny Doe (STRING) IN 1 AT 1388073617954000 | | 2013-12-26T10:00:23.859-06:00 | ADD name AS Jonathan Doe (STRING) IN 1 AT 1388073623859000 | | 2013-12-26T10:00:28.412-06:00 | ADD name AS J. Doe (STRING) IN 1 AT 1388073628412000 | | 2013-12-26T10:00:38.249-06:00 | ADD age AS 30 (INTEGER) IN 1 AT 1388073638249000 | | 2013-12-26T10:00:43.338-06:00 | ADD age AS 30.5 (FLOAT) IN 1 AT 1388073643338000 | | 2013-12-26T10:00:49.834-06:00 | ADD age AS 30 (STRING) IN 1 AT 1388073649834000 | | 2013-12-26T10:00:54.599-06:00 | ADD age AS true (BOOLEAN) IN 1 AT 1388073654599000 | | 2013-12-26T10:01:00.678-06:00 | REMOVE age AS true (BOOLEAN) IN 1 AT 1388073660678000 | | 2013-12-26T10:01:38.086-06:00 | ADD baz AS 0 (INTEGER) IN 1 AT 1388073698086000 | | 2013-12-26T10:01:38.089-06:00 | ADD baz AS 1 (INTEGER) IN 1 AT 1388073698089000 | | 2013-12-26T10:01:38.090-06:00 | ADD baz AS 2 (INTEGER) IN 1 AT 1388073698090000 | | 2013-12-26T10:01:38.092-06:00 | ADD baz AS 3 (INTEGER) IN 1 AT 1388073698092000 | | 2013-12-26T10:01:38.093-06:00 | ADD baz AS 4 (INTEGER) IN 1 AT 1388073698093000 | | 2013-12-26T10:01:50.478-06:00 | REMOVE baz AS 0 (INTEGER) IN 1 AT 1388073710478000 | | 2013-12-26T10:01:50.481-06:00 | REMOVE baz AS 1 (INTEGER) IN 1 AT 1388073710481000 | | 2013-12-26T10:01:50.482-06:00 | REMOVE baz AS 2 (INTEGER) IN 1 AT 1388073710482000 | | 2013-12-26T10:01:50.483-06:00 | REMOVE baz AS 3 (INTEGER) IN 1 AT 1388073710483000 | | 2013-12-26T10:01:50.484-06:00 | REMOVE baz AS 4 (INTEGER) IN 1 AT 1388073710484000 | | 2013-12-26T10:01:50.486-06:00 | ADD baz AS 6 (INTEGER) IN 1 AT 1388073710486000 | | 2013-12-26T10:03:46.305-06:00 | ADD count AS 1 (INTEGER) IN 1 AT 1388073826305000 | +--------------------------------------------------------------------------------------------+ ' in 17 ms
Audit age in record 1.
cash$ concourse.audit "age", 1 Returned ' +---------------------------------------------------------------------------------------+ | DateTime | Revision | +---------------------------------------------------------------------------------------+ | 2013-12-24T09:40:21.424-06:00 | ADD age AS 30 (INTEGER) IN 1 AT 1387899621424000 | | 2013-12-24T09:40:31.651-06:00 | ADD age AS 30.5 (STRING) IN 1 AT 1387899631651000 | | 2013-12-24T09:40:37.574-06:00 | ADD age AS 30 (STRING) IN 1 AT 1387899637574000 | | 2013-12-24T09:40:41.976-06:00 | ADD age AS true (BOOLEAN) IN 1 AT 1387899641976000 | | 2013-12-24T09:40:49.429-06:00 | REMOVE age AS true (BOOLEAN) IN 1 AT 1387899649429000 | +---------------------------------------------------------------------------------------+ ' in 17 ms
Revert
The revert function atomically returns a key in a record to a previous state (specified by a Timestamp).
Revert baz in record 1 to the timestamp when 2 was added.
cash$ it = concourse.audit(1).keySet().iterator(); for(int i = 0; i < 11; i++){it.next(); }; t1 = it.next(); // save the relevant timestamp to t1
cash$ concourse.revert "baz", 1, t1
Completed in 46 ms
concourse.fetch "baz", 1
Returned '[0, 1, 2]' in 16 ms
Historical Operations
Concourse allows you to describe, fetch, verify, and find data from the past.
Describe record 1 yesterday.
cash$ concourse.describe 1, time("yesterday")
Returned '[]' in 23 ms
Describe record 1 at the timestamp before the first value for baz was added.
cash$ it = concourse.audit(1).keySet().iterator(); for(int i = 0; i < 8; i++){it.next(); }; t1 = it.next(); // save the relevant timestamp to t1
cash$ concourse.describe 1, time((t1.getMicros() - 1))
Returned '[name, age]' in 25 ms
Fetch name from record 1 at the timestamp when you added name as Johnny Doe.
cash$ it = concourse.audit(1).keySet().iterator(); for(int i = 0; i < 1; i++){it.next(); }; t1 = it.next(); // save the relevant timestamp to t1
cash$ concourse.fetch "name", 1, t1
Returned '[John Doe, Johnny Doe]' in 20 ms
Find count greater than 50 at the timestamp when count as 50 was added
cash$ t1 = concourse.audit(50).keySet().iterator().next(); // save the relevant timestamp to t1 cash$ concourse.find "count", gt, 50, t1 Returned '[]' in 106 ms
Find count greater than 50 at the timestamp when count as 500 was added.
cash$ t1 = concourse.audit(500).keySet().iterator().next(); // save the relevant timestamp to t1 cash$ concourse.find "count", gt, 50, t1 Returned '[51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500]' in 252 ms
Verify age as true in record 1 at the timestamp when you added age as true in record 1.
cash$ it = concourse.audit("age", 1).keySet().iterator(); for(int i = 0; i < 3; i++){it.next(); }; t1 = it.next(); // save the relevant timestamp to t1
cash$ concourse.verify "age", true, 1, t1
Returned 'true' in 14 ms
Searching
TODO: finishme
Concourse automatically indexes text data for full text substring search
cash$ strings = ["The Cat in the Hat", "Green Eggs and Ham", "Horton Hears a Who", "The Cat in the Hat Comes Back", "Scrambled Eggs Super"]
Returned '[The Cat in the Hat, Green Eggs and Ham, Horton Hears a Who, The Cat in the Hat Comes Back, Scrambled Eggs Super]' in 340 ms
cash$ for(int i = 0; i < 1000; i++){ value = strings[i % strings.size()]; concourse.add "title", value, i}
Completed in 487 ms
cash$ concourse.search "title", "eggs"
Returned '[956, 959, 954, 889, 949, 891, 951, 944, 946, 894, 941, 936, 939, 994, 934, 929, 996, 999, 931, 986, 926, 984, 924, 991, 989, 921, 979, 919, 976, 916, 914, 981, 971, 911, 969, 909, 974, 906, 904, 961, 901, 899, 966, 964, 896, 1, 4, 6, 9, 11, 14, 16, 19, 21, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 111, 114, 116, 119, 121, 124, 126, 129, 131, 134, 136, 139, 141, 144, 146, 149, 151, 154, 156, 159, 161, 164, 166, 169, 171, 174, 176, 179, 181, 184, 186, 189, 191, 194, 196, 199, 201, 204, 206, 209, 211, 214, 216, 219, 221, 224, 226, 229, 231, 234, 236, 239, 241, 244, 246, 249, 251, 254, 256, 259, 261, 264, 266, 269, 271, 274, 276, 279, 281, 284, 286, 289, 291, 294, 296, 299, 301, 304, 306, 309, 311, 314, 316, 319, 321, 324, 326, 329, 331, 334, 336, 339, 341, 344, 346, 349, 351, 354, 356, 359, 361, 364, 366, 369, 371, 374, 376, 379, 381, 384, 386, 389, 391, 394, 396, 399, 401, 404, 406, 409, 411, 414, 416, 419, 421, 424, 426, 429, 431, 434, 436, 439, 441, 444, 446, 449, 451, 454, 456, 459, 461, 464, 466, 469, 471, 474, 476, 479, 481, 484, 486, 489, 491, 494, 496, 499, 501, 504, 506, 509, 511, 514, 516, 519, 521, 524, 526, 529, 531, 534, 536, 539, 541, 544, 546, 549, 551, 554, 556, 559, 561, 564, 566, 569, 571, 574, 576, 579, 581, 584, 586, 589, 591, 594, 596, 599, 601, 604, 606, 609, 611, 614, 616, 619, 621, 624, 626, 629, 631, 634, 636, 639, 641, 644, 646, 649, 651, 654, 656, 659, 661, 664, 666, 669, 671, 674, 676, 679, 681, 684, 686, 689, 691, 694, 696, 699, 701, 704, 706, 709, 711, 714, 716, 719, 721, 724, 726, 729, 731, 734, 736, 739, 741, 744, 746, 749, 751, 754, 756, 759, 761, 764, 766, 769, 771, 774, 776, 779, 781, 784, 786, 789, 791, 794, 796, 799, 801, 804, 806, 809, 811, 814, 816, 819, 821, 824, 826, 829, 831, 834, 836, 839, 841, 844, 846, 849, 851, 854, 856, 859, 861, 864, 866, 869, 871, 874, 876, 879, 881, 884, 886]' in 86 ms
Next Steps
Try the TwitterCLI Exercise.