Mount block device

How to mount a new block device (CentOS)

# Check current status: shows no partition info and disk ID=0x0

sudo fdisk -l

Disk /dev/xvdc: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

# Create a partition. Answer prompts as indicated
sudo fdisk /dev/xvdc
n # new partition
p # primary
1 # partition no.
<enter> # accept start cyclinder default
<enter> # accept end cyclinder default (take whole device!)
w # write partition

# Verify. fdisk should show new partition info
sudo fdisk -l

Disk /dev/xvdc: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xb7719266

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdc1               1       26108   209712478+  83  Linux

# Format new device
sudo mkfs -t ext3 /dev/xvdc1

# Mount it
[ ! -d /data ] && sudo mkdir /data
sudo mount /dev/xvdc1 /data

# Make permanent
sudo nano /etc/fstab
# edit to add:
/dev/xvdc1            /data                    ext3    defaults,noatime,barrier=0      1 1

Generate Elasticsearch test documents

for i in $(seq 1 100); do echo curl -XPUT \''http://roadrunner2:9200/test-001/doc/'${i}''\' -d \''{"sentence":"'$(head -c 56 < /dev/urandom | base64)'"}'\'; done | bash

Install iPerf on CentOS

yum -y install yum-plugin-priorities
sed -i -e "s/\]$/\]\npriority=1/g" /etc/yum.repos.d/CentOS-Base.repo
yum -y install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sed -i -e "s/\]$/\]\npriority=5/g" /etc/yum.repos.d/epel.repo
yum --enablerepo=epel -y install iperf3   # install from EPEL

Sensei

Storm:

     http://www.michael-noll.com/tutorials/

Clojure:

     http://ben.vandgrift.com/2013/03/13/clojure-hello-world.html

     http://www.braveclojure.com/getting-started/

     http://jrheard.tumblr.com/post/40024238467/getting-started-with-clojure

Distrosys:

     http://aphyr.com/

Clojure and Storm

$ lein repl

(require '[clj-http.client :as client])
(client/get "http://asterdc:8080/api/v1/getfile?name=stream.10.log")

(def stream (client/get "http://node1:8080/api/v1/getfile?name=stream.10.log"))

(require '[clojure.data.json :as json])

user=> (type stream)
clojure.lang.PersistentArrayMap

user=> (keys stream)
(:orig-content-encoding :trace-redirects :request-time :status :headers :body)

user=> (count (:body stream)) ;; count characters in body
20279

user=> (get (.split (:body stream) "\n") 0) ;; get first body from array
"{\"id\":\"tag:search.twitter.com,2005:178204609083473921\",\"body\":\"RT @mikeanikevans: I am such a queer\",\"verb\":\"share\",\"link\":\"http://twitter.com/gthang42069/statuses/178204609083473921\",\"generator\":...{\"klout_score\":27}}\r"

user=> (get (json/read-str (get (.split (:body stream) "\n") 9)) "body")
"@MEMOZ809 No se :("

user=> (keys (json/read-str (get (.split (:body stream) "\n") 9)))
("inReplyTo" "body" "postedTime" "twitter_entities" "gnip" "provider" "generator" "actor" "objectType" "link" "id" "verb" "object")

# run topology from command line (no Storm runtime)

$ lein run -m fiat.topology/run!

---

lein clear

lein compile

lein jar

To run on a local cluster:

```bash
lein run -m fiat.topology/run!
# OR
lein run -m fiat.topology/run! debug false workers 10
```

To run on a distributed cluster:

```bash
lein uberjar
# copy jar to nimbus, and then on nimbus:
bin/storm jar path/to/uberjar.jar fiat.TopologySubmitter workers 30 debug false
```

Install local Storm jars on master node

```

/usr/lib/maven/bin/mvn deploy:deploy-file  -Dfile=/usr/lib/storm/lib/storm-core-0.9.1.2.1.2.0-402.jar -DartifactId=storm -DgroupId=local -Dversion=0.9.1.2.1.2.0-402 -Dpackaging=jar -Durl=file:repo

```

Install lein-localrepo per https://github.com/kumarshantanu/lein-localrepo

lein clean; lein deps; lein compile; lein uberjar

Clojure and HBase

(println (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))) ;; get class path in REPL

# in REPL

(require '[clojure-hbase.core :as hbase])

(import [org.apache.hadoop.hbase HBaseConfiguration HConstants KeyValue])

(import [org.apache.hadoop.hbase.client HTablePool Get Put Delete Scan Result RowLock HTableInterface])

(hbase/set-config (hbase/make-config {

     :zookeeper.znode.parent "/hbase-unsecure" 

     :hbase.zookeeper.property.clientPort "2181"

     :hbase.cluster.distributed "true"

     :hbase.zookeeper.quorum "hdp005-3,hdp005-21,hdp005-23"

}

))

(hbase/table "tweets-test")

---

create 'tweets_test', 'cf'

put "tweets_test", "newkey1", 'cf', 'somevalue'

hbase(main):007:0> scan "tweets_test"
ROW                             COLUMN+CELL
newkey1                        column=cf:, timestamp=1405693122884, value=somevalue

Clojure cider, lein

lein repl :start :port 10010

(use 'clojure.repl 'clojure.pprint)

(setq nrepl-popup-stacktraces-in-repl nil)
(setq nrepl-auto-select-error-buffer nil)

Zoom

You can use `C-x C-+’ and ‘C-x C--’ (‘text-scale-adjust’) to increase or decrease the buffer text size (`C-+’ or ‘C--’ to repeat).

Very important: add cider plugin in project.clj

(defproject lucy "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [
                 [org.clojure/clojure "1.6.0"]
                 [clucy "0.4.0"]
                 ]
  :plugins [[cider/cider-nrepl "0.6.0"]]
  )

In CIDER

C-c M-n: Set REPL namespace from buffer

C-x C-e: Eval preceding line

C-c C-k: compile buffer

C-x C-c: eval region

C-c C-c: abort eval

M-: Read a single Emacs Lisp expression in the minibuffer, evaluate it, and print the value in the echo area (eval-expression).

C-x h selects the entire buffer.

C-M-\ reindents the selected region.

C-M-@ select s-expression

M-. Jump to the definition of a symbol. If invoked with a prefix argument, or no symbol is found at point, prompt for a symbol.

M-@ (mark-word) puts the mark at the end of the next word

Use M-< to move to the beginning of the buffer, and M-> to move to the end

C-v to scroll down, and M-v to scroll up

M-d Kill up to the end of a word (kill-word).

M-\ Delete spaces and tabs around point (delete-horizontal-space)

M-<SPC> Delete spaces and tabs around point, leaving one space (just-one-space)

M-^ Join two lines by deleting the intervening newline, along with any indentation following it (delete-indentation)

C-t Transpose two characters (transpose-chars).

M-t Transpose two words (transpose-words).

C-M-t Transpose two balanced expressions (transpose-sexps).

C-x C-t Transpose two lines (transpose-lines).

M-x auto-revert-tail-mode (tail a file)

You can hide the *nrepl-connection* and *nrepl-server* buffers from appearing in some buffer switching commands like switch-to-buffer(C-x b) like this:
(setq nrepl-hide-special-buffers t)

---

M-% string <RET> newstring <RET>

Replace some occurrences of string with newstring.

C-M-% regexp <RET> newstring <RET>

Replace some matches for regexp with newstring.

---

<SPC>

to replace the occurrence with newstring.

<DEL>

to skip to the next occurrence without replacing this one.

, (Comma)

to replace this occurrence and display the result. You are then asked for another input character to say what to do next. Since the replacement has already been made, <DEL> and <SPC> are equivalent in this situation; both move to the next occurrence.You can type C-r at this point (see below) to alter the replaced text. You can also type C-x u to undo the replacement; this exits the query-replace, so if you want to do further replacement you must use C-x <ESC> <ESC> <RET> to restart (see Repetition).

<RET>

to exit without doing any more replacements.

. (Period)

to replace this occurrence and then exit without searching for more occurrences.

!

to replace all remaining occurrences without asking again.

Y (Upper-case)

to replace all remaining occurrences in all remaining buffers in multi-buffer replacements (like the Dired `Q' command which performs query replace on selected files). It answers this question and all subsequent questions in the series with "yes", without further user interaction.

N (Upper-case)

to skip to the next buffer in multi-buffer replacements without replacing remaining occurrences in the current buffer. It answers this question "no", gives up on the questions for the current buffer, and continues to the next buffer in the sequence.

^

to go back to the position of the previous occurrence (or what used to be an occurrence), in case you changed it by mistake or want to reexamine it.

---

Print object members

(doseq [m (.getMethods (type index))] (println m))

(use 'clojure.reflect 'clojure.pprint)

(pprint (reflect "hello"))

{:bases
 #{java.io.Serializable java.lang.Comparable java.lang.Object
   java.lang.CharSequence},
 :flags #{:public :final},
 :members
 #{{:name valueOf,
    :return-type java.lang.String,
    :declaring-class java.lang.String,...

List classpath

(defn classpath []

(seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))))

Expand macros

(macroexpand '(time (print "timing")))

;; or better yet

(clojure.pprint/pprint (macroexpand '(time 1)))

List members in namespace

(require '[clojure-hbase.core :as hbase])

(dir clojure-hbase.core)

Show function source

user=> (source hbase/table)
(defn table
  "Gets an HTable from the open HTablePool by name."
  [table-name]
  (io!
   (.getTable (htable-pool) (to-bytes table-name))))

Find elements by regex match in collection

(filter #(re-find #"zoo" (key %)) (seq (hbase/make-config nil)))

clojure.core/all-ns
  Returns a sequence of all namespaces.

List files

(take 10 (file-seq (clojure.java.io/file ".")))

Clojure and Zookeeper

Add [zookeeper-clj "0.9.1"] to dependencies

lein repl

(require '[zookeeper :as zk])

(.getChildren client "/" false) ;; watcher=false

#<ArrayList [hbase-unsecure, storm, zookeeper]>

(def ZK_HOSTS "127.0.0.1:2181,node1,node2,node3)

(def ZK_ROOT "/twitter-demo")(def client (zk/connect ZOOKEEPER_HOSTS :watcher (fn [event] (println event))))

(zk/create client ZK_ROOT :persistent? false)

(def version (:version (zk/exists client ZK_ROOT)))

(zk/create client ZK_ROOT :data (.getBytes "123143kdjkds") :persistent? false)

Emacs

open
emacs& RET     or click a graphical icon
emacs -nw RET     start in terminal, not in a windowing system

cancel

C-g     a command

M-g g    Go to line #

Text size
Shrink/Grow: C - - / C - + 

Copy & Paste

C-<space>
    Start selection
C-w
    Cut
M-w
    Copy
C-y
    Paste

12.1.2 Killing by Lines

C-k

Kill rest of line or one or more lines (kill-line).

C-S-backspace

Kill an entire line at once (kill-whole-line)

M-: evaluate function in minibuffer

(vi yy): C-a (beginning) C-spc (highlight) C-e (end) C-y (yank)

M-<Move to the top of the buffer (beginning-of-buffer). With numeric argument n, move to n/10 of the way from the top.
M->Move to the end of the buffer (end-of-buffer). 

15.10.4 Query Replace

M-% string <RET> newstring <RET>

Replace some occurrences of string with newstring. 

C-M-% regexp <RET> newstring <RET>

Replace some matches for regexp with newstring.

26.3.2 Indenting Several Lines

C-M-q

Reindent all the lines within one parenthetical grouping. 

C-u <TAB>

Shift an entire parenthetical grouping rigidly sideways so that its first line is properly indented. 

M-x indent-code-rigidly

Shift all the lines in the region rigidly sideways, but do not alter lines that start inside comments and strings.

C-u 4 M-x indent-rigidly to indent the region by four spaces, C-u -4 M-x indent-rigidly to remove four spaces

Also, in Python mode, with a marked region, use C-c > and C-c <

Displaying in Another Window

C-x 4 is a prefix key for commands that select another window (splitting the window if there is only one) and select a buffer in that window. Different C-x 4commands have different ways of finding the buffer to select.

C-x 4 b bufname RET

Select buffer bufname in another window. This runs switch-to-buffer-other-window.

C-x 4 f filename RET

Visit file filename and select its buffer in another window. This runs find-file-other-window. See section Visiting Files.

C-x 4 d directory RET

Select a Dired buffer for directory directory in another window. This runs dired-other-window. See section Dired, the Directory Editor.

C-x 4 m

Start composing a mail message in another window. This runs mail-other-window, and its same-window version is C-x m (see section Sending Mail).

C-x 4 .

Find a tag in the current tag table in another window. This runs find-tag-other-window, the multiple-window variant of M-. (see section Tag Tables).

Deleting and Rearranging Windows

C-x 0

Get rid of the selected window (kill-window). That is a zero.

C-x 1

Get rid of all windows except the selected one (delete-other-windows).

C-x ^

Make the selected window taller, at the expense of the other(s) (enlarge-window).

C-x }

Widen the selected window (enlarge-window-horizontally).

Linux handling of sockets and open files

# kill all processes with open TCP sockets
kill -9 `lsof | grep ESTABLISHED | awk '{print $2}'`

# kill processes by port number of the TCP socket
fuser -k 22/tcp

Teradata: Normalize BTEQ output

bteq <<EOF | 
awk 'BEGIN { 
   start=0; 
} 
{ 
   if (start==1) { 
      if (/^$/) { 
         exit; 
      } else { 
         print; 
      } 
   } 
   if (/^---/) { 
      start=1; 
   } 
}'
.logon host/user,password
select top 3 tablename
from dbc.tables
where databasename = 'myDB'
and tablekind='T'
order by 1
;
.logoff
.exit
EOF

bash: randomize order of arguments

n=$#
c=0
declare -a s
for r in $(seq 1 $n)
do
while [ -z $i ] || [ $((${s[$i]})) -eq 1 ] && [ $c -lt $n ]
do
u=$(od -N2 -tu2 -An /dev/urandom)
i=$(( $u % $n + 1))
done
s[$i]=1
c=$(($c + 1))
echo $i
i=''
done

BASH: Random text file sample (linewise)

num=$(wc -l < file.txt)
for i in $(seq 1 1000)
do
   n=$(( $RANDOM * $RANDOM % $num ))
   sed -n "${n}p" file.txt >> /tmp/file.txt.sample
done