Docker lessons from the front lines

• Be careful when mapping host volumes: filesystem types matter greatly in Docker, and something as simple as mapping /tmp or /dev from the host can result in obscure errors, e.g. Error response from daemon: Cannot start container xxxxxxx : [8] System error: no such file or directory

• Default log file locations for the Docker daemon vary by distribution, typically determined by the init system:
• /var/log/upstart/docker.log (Debian/Ubuntu)
• /var/log/docker.log (CentOS)

• Start Docker with debug logging.
• Directly from command line: 
• sudo DOCKER_OPTS='--log-level=debug' service docker start
• Also possible through settings in /etc/default/docker.conf

Linux resource monitoring tools

Command line tools for monitoring system resource utilization:

• top
• htop (visual version of top)
• iotop (disk I/O by process)
• iftop (network I/O by host)

Run NPM behind a proxy

NPM does not obey the proxy configuration set in $http_proxy and $https_proxy as curl and other common tools. It is instead necessary to define NPM-specific settings as follows.
 
npm config set strict-ssl false
npm config set registry "http://registry.npmjs.org/"

# Adjust username, password and proxy endpoint as needed
npm set proxy $http_proxy
# Also for HTTPS: pay attention to protocol 
npm set https-proxy https://<yourproxy>

These last two commands simply write the specified values to their ultimate location: ~/.npmrc

If you need to specify credentials and/or a port, use:

username:password@yourproxy:port, e.g.
proxyuser:pwd123@proxy.mydomain.com:3128
 

Reliably determine a script's directory path in bash

cd $(dirname $0); DIR=$PWD; cd - >/dev/null

NPM without sudo

In development environments, access to install NPM global packages under the standard /usr path if often restricted. Instead of using sudo for global package installation, which is rarely the desired solution in development, it is possible to instruct NPM to use a directory owned by the logon user as follows:

mkdir ~/npm
npm config set prefix ~/npm

After this, installing a package globally, such as:

npm install -g async

will result int the library files being placed under the configured location (~/npm in this example).

Awk: generate random sentences

cat /usr/share/dict/words | 
  grep -v "'" | 
  shuf | 
  awk -v wordCount=100 -v wordsPerSentence=5 '
    BEGIN {i=0} 
    (NR<wordCount) {
      if (i<wordsPerSentence) {
        a[i]=$0; 
        i++
      } else {
        for (j=0;j<length(a);j++) {
          printf ("%s ",a[j])
      }
      print
      i=0
    }
  }'

Stop Firefox from turning localhost into www.localhost.com

In URL bar, enter about:config

Search for setting browser.fixup.alternate.enabled and set to false

Upgrade git from upstream

(Debian)

As of 2015, most Debian variants provide git 1.9.4. To upgrade to the 2.x line, use the Git core PPA as follows:

sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

$ git --version
git version 2.5.0

Install emacs from source

mkdir /tmp/emacs

cd /tmp/emacs

wget https://ftp.gnu.org/pub/gnu/emacs/emacs-24.4.tar.xz 

tar xvf emacs-24.4.tar.xz 

cd emacs-24.4 

./configure --prefix=/usr        \
            --without-gnutls     \
            --localstatedir=/var &&
make

if make succeeds, you can test the result by running src/emacs -Q 

sudo make install && sudo chown -v -R root:root /usr/share/emacs/24.4

For a desktop launcher:

vim cat <<EOF > /usr/share/applications/emacs.desktop 

[Desktop Entry]
Version=1.0
Name=Emacs-24
Exec=/usr/local/bin/emacs
Terminal=false
Icon=emacs
Type=Application
Categories=IDE
[NewWindow Shortcut Group]
Name=New Window

EOF

Desktop .bashrc

case $- in
    *i*) ;;
      *) return;;
esac

shopt -s histappend
HISTSIZE=1000
HISTFILESIZE=2000
HISTCONTROL=ignoreboth

shopt -s checkwinsize

if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

case "$TERM" in
    xterm|xterm-color|*-256color) color_prompt=yes;;
esac

force_color_prompt=yes

case "$TERM" in
xterm*|rxvt*)
    LIGHT_GRAY="\[\033[0;37m\]"; BLUE="\[\033[1;36m\]"; RED="\[\033[0;31m\]"; LIGHT_RED="\[\033[1;31m\]";
    GREEN="\[\033[0;32m\]"; WHITE="\[\033[1;37m\]"; LIGHT_GRAY="\[\033[0;37m\]"; YELLOW="\[\033[1;33m\]";
    function parse_git_branch {
        git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ \(\1\)/';
    }
    function parse_git_status {
        git status 2> /dev/null | sed -e '/(working directory clean)$/!d' | wc -l;
    }
    function check_git_changes {
        # tput setaf 1 = RED, tput setaf 2 = GREEN
        [ `parse_git_status` -ne 1 ] && tput setaf 1 || tput setaf 2
    }
    export PS1="($(whoami)@$(hostname)) \[$(tput bold)\]\[$(tput setaf 2)\]$(date +%F\|%R) $YELLOW\w\[\$(check_git_changes)\]\$(parse_git_branch)$LIGHT_GRAY\n $ "
    export PS4='+(${BASH_SOURCE}:${LINENO})'
    ;;
*)
    ;;
esac

set -o emacs
umask 002
export HISTTIMEFORMAT="%F %T "
shopt -s histappend

export TZ='/usr/share/zoneinfo/US/Eastern'
export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"

alias l='ls -lF'
alias ll='ls -alF'

if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre

Export Gimp layers as separate files

Install plugin at https://github.com/khalim19/gimp-plugin-export-layers

Extract the export_layers.py file and the export_layers directory to [home directory]/.gimp-2.8/plug-ins.

If necessary, make the export_layers.py file executable, e.g. from the terminal:

chmod +x "export_layers.py"

Collate PDF documents

pdftk $(ls file*.pdf|sort) cat output combined.pdf

Tech links

• Vert.x: Reactive server for the JVM
• Wildfly, Apiman, Keycloak

Network traffic monitoring with tcpdump

tcpdump is a command-line network traffic analyzer. It allows capture of network packets from a host and subsequent filtering and display.

Sample applications

Show traffic into port 2003 from a given host

sudo tcpdump -A -nn 'port 2003 and src host 10.60.35.49'

Must run as root (sudo)
-A print ASCII payload
-nn numeric IP and ports

Snippets

# Sudo a long command line
sudo bash -c "
# Get 16th field
cut -f16 -d\| /data/sample.csv | 
# Render unicode as ASCII
iconv -f utf8 -t ascii//TRANSLIT | 
# Drop empty lines
sed '/^$/d' | 
# Filter for lines longer than 30 characters
awk 'length(\$0) > 30' > /data/sample.out"

NUMA, affinity and OS tuning

NUMA tuning and affinity (RHEL)
NUMA and optimization (Elasticsearch, Riak)

IRC cheat sheet

Register IRC nick

/msg NickServ REGISTER password youremail@example.com

Identify to primary IRC account:

/msg NickServ IDENTIFY account password

Get nick info

/msg NickServ INFO account

Reset password

/msg NickServ SET PASSWORD mynewpassword

Reset command hash

To flush the hash table used by the shell to remember the full path names of executable files:

hash -r

git recipes

Git: prune local branches without remotes

git branch -vv | grep ': gone]' | awk '{print $1}' | xargs git branch -D 

Cache git credentials (Linux)

While it is possible to store git credentials locally via .netrc, that requires storing them in plain text. Even with 400 permissions, it is an unnecessary risk. I have read references about using GPG for encryption within this file, which increases security, but introduces a new dependency. When changing hosts frequently, local dependencies become highly impractical. Instead, to cache the credentials in memory with an expiration timeout, one hour, for example:

git config --global credential.helper 'cache --timeout=3600'

Fix brightness controls in Ubuntu 14

In some instances of Ubuntu 14, brightness control keys do not work. To correct this, first determine the type  of video card adapter controlling the display backlight; use:

$ ls /sys/class/backlight
intel_backlight

sudo vim /usr/share/X11/xorg.conf.d/20-intel.conf

Add the following lines to this file:

Section "Device"

        Identifier  "card0"
        Driver      "intel"
        Option      "Backlight"  "intel_backlight"
        BusID       "PCI:0:2:0"
EndSection

Save it. Log out and log in back. The brightness control should be working through function keys now:

Setting up Scalding

Twitter does not publish Scalding binaries. To set up Scalding locally, the following is needed:

1. Clone: git clone https://github.com/twitter/scalding.git
2. sbt assembly
3. If necessary, adjust sbt-assembly version accoridng to Scala/sbt versions (use sbt about). Under project/plugins.sbt, e.g.:

-addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.10.2")
+addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
 
4. sbt publishLocal