How to start a hadoop ec2 cluster using crane.

Workflow:
-def conf map using conf fn
-def new Jec2 class, stores creds
-launch hadoop cluster

Configuration maps:

creds.clj

{:key "AWS-KEY"
 :secretkey "AWS-SECRET-KEY"
 :private-key-path "/path/to/private-key"
 :key-name "key-name"}

conf.clj

{:image "ami-"
 :instance-type :m1.large
 :group "ec2-group"
 :instances int
 :instances int
 :creds "/path/to/creds.clj-dir"
 :push ["/source/path" "/dest/path/file"
        "/another/source/path" "/another/dest/path/file"]
 :hadooppath "/path/to/hadoop"          ;;used for remote pathing
 :hadoopuser "hadoop-user"              ;;remote hadoop user
 :mapredsite "/path/to/local/mapred"    ;;used as local templates
 :coresite "/path/to/local/core-site"
 :hdfssite "/path/to/local/hdfssite"}

Example:

;;read in config "aws.clj"
crane.cluster> (def my-conf (conf "/path/to/conf.clj-dir"))

;;create new Jec2 class
crane.cluster> (def ec (ec2 (creds "/path/to/creds.clj-dir")))

;;start cluster 
(launch-hadoop-cluster ec my-conf)

To build:

1. Download and install leiningen http://github.com/technomancy/leiningen
2. Setup jclouds maven snapshot at http://jclouds.rimuhosting.com/maven2/snapshots
3. $ lein deps
4. $ lein install
5. $ java -cp `ls lib/*|xargs echo|sed 's/ /:/g'`:classes jline.ConsoleRunner clojure.main

core libs for compute cloud, blobstore, and ssh.

supported compute clouds: ec2, rackspace, rimuhosting, vcloud, terremark, hosting.com
supported blobstores: s3, rackspace, azure, atmos online

build upon the core is remote repl and cluster capabilities.

in a couple cases, we use an erlang convention of the ! syntax representing "reomote" (on another process) execution.
the first argument to ! in erlang is the pid, wheras in our api, it is an open ssh session or socket.
there is no ! operator or fn, we just use the ! suffix convention to communicate remote execution.

remote shell execution.
(sh! session "tar -xzf repl.tar.gz")

remote repl evaluation.
eval! 
(eval! socket (execute (workflow some-cascading-workflow)))

Cloud Examples:

Here's a quick example of how to view compute and blob resources.

(ns crane.jclouds
  (:use crane.blobstore)
  (:use crane.compute)
  (:use clojure.contrib.pprint)
)

 (def user  "rackspace_username")
 (def password "rackspace_password")

 (def compute-name "cloudservers")
 (def compute (compute-context compute-name user password))
 (def blobstore-name "cloudfiles")
 (def blobstore (blobstore-context blobstore-name user password ))

 (pprint (images compute))
 (pprint (nodes compute))
 (pprint (sizes compute))

 (pprint (containers blobstore))
 (pprint (blobs blobstore "your_container_name" )) 


TODO:
the persistent shell session using jsch ChannelShell is shaky at best, although exec works fine.
eval! may be simpler if we could use LineNumberingPushbackReader see comment in remote_repl.clj
stuff to poll and pull info from hadoop tracker url into repl