More jq magic for JSON wrangling

Count number of JSON elements in an embedded array

Data out of an Elasticsearch aggregation result set: 

{
  "aggregations": {
    "flightdestination": {
      "buckets": [
        {
          "sentiment": {
            "value": 2.1706734237483767
          },
          "doc_count": 6931,
          "key": "newark"
        },
        {
          "sentiment": {
            "value": 2.17875893247776
          },
          "doc_count": 6857,
          "key": "houston"

        },

...

curl -s 'http://10.60.35.34:9200/1$1_0/sentence/_search' -XGET --data "$(cat ~/query.json)" | jq '.aggregations.flightdestination.buckets | length'

Docker lessons from the front lines

• Be careful when mapping host volumes: filesystem types matter greatly in Docker, and something as simple as mapping /tmp or /dev from the host can result in obscure errors, e.g. Error response from daemon: Cannot start container xxxxxxx : [8] System error: no such file or directory

• Default log file locations for the Docker daemon vary by distribution, typically determined by the init system:
• /var/log/upstart/docker.log (Debian/Ubuntu)
• /var/log/docker.log (CentOS)

• Start Docker with debug logging.
• Directly from command line: 
• sudo DOCKER_OPTS='--log-level=debug' service docker start
• Also possible through settings in /etc/default/docker.conf

Linux resource monitoring tools

Command line tools for monitoring system resource utilization:

• top
• htop (visual version of top)
• iotop (disk I/O by process)
• iftop (network I/O by host)

Run NPM behind a proxy

NPM does not obey the proxy configuration set in $http_proxy and $https_proxy as curl and other common tools. It is instead necessary to define NPM-specific settings as follows.
 
npm config set strict-ssl false
npm config set registry "http://registry.npmjs.org/"

# Adjust username, password and proxy endpoint as needed
npm set proxy $http_proxy
# Also for HTTPS: pay attention to protocol 
npm set https-proxy https://<yourproxy>

These last two commands simply write the specified values to their ultimate location: ~/.npmrc

If you need to specify credentials and/or a port, use:

username:password@yourproxy:port, e.g.
proxyuser:pwd123@proxy.mydomain.com:3128
 

Reliably determine a script's directory path in bash

cd $(dirname $0); DIR=$PWD; cd - >/dev/null

NPM without sudo

In development environments, access to install NPM global packages under the standard /usr path if often restricted. Instead of using sudo for global package installation, which is rarely the desired solution in development, it is possible to instruct NPM to use a directory owned by the logon user as follows:

mkdir ~/npm
npm config set prefix ~/npm

After this, installing a package globally, such as:

npm install -g async

will result int the library files being placed under the configured location (~/npm in this example).

Awk: generate random sentences

cat /usr/share/dict/words | 
  grep -v "'" | 
  shuf | 
  awk -v wordCount=100 -v wordsPerSentence=5 '
    BEGIN {i=0} 
    (NR<wordCount) {
      if (i<wordsPerSentence) {
        a[i]=$0; 
        i++
      } else {
        for (j=0;j<length(a);j++) {
          printf ("%s ",a[j])
      }
      print
      i=0
    }
  }'

Stop Firefox from turning localhost into www.localhost.com

In URL bar, enter about:config

Search for setting browser.fixup.alternate.enabled and set to false

Upgrade git from upstream

(Debian)

As of 2015, most Debian variants provide git 1.9.4. To upgrade to the 2.x line, use the Git core PPA as follows:

sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

$ git --version
git version 2.5.0

Install emacs from source

mkdir /tmp/emacs

cd /tmp/emacs

wget https://ftp.gnu.org/pub/gnu/emacs/emacs-24.4.tar.xz 

tar xvf emacs-24.4.tar.xz 

cd emacs-24.4 

./configure --prefix=/usr        \
            --without-gnutls     \
            --localstatedir=/var &&
make

if make succeeds, you can test the result by running src/emacs -Q 

sudo make install && sudo chown -v -R root:root /usr/share/emacs/24.4

For a desktop launcher:

vim cat <<EOF > /usr/share/applications/emacs.desktop 

[Desktop Entry]
Version=1.0
Name=Emacs-24
Exec=/usr/local/bin/emacs
Terminal=false
Icon=emacs
Type=Application
Categories=IDE
[NewWindow Shortcut Group]
Name=New Window

EOF

Desktop .bashrc

case $- in
    *i*) ;;
      *) return;;
esac

shopt -s histappend
HISTSIZE=1000
HISTFILESIZE=2000
HISTCONTROL=ignoreboth

shopt -s checkwinsize

if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

case "$TERM" in
    xterm|xterm-color|*-256color) color_prompt=yes;;
esac

force_color_prompt=yes

case "$TERM" in
xterm*|rxvt*)
    LIGHT_GRAY="\[\033[0;37m\]"; BLUE="\[\033[1;36m\]"; RED="\[\033[0;31m\]"; LIGHT_RED="\[\033[1;31m\]";
    GREEN="\[\033[0;32m\]"; WHITE="\[\033[1;37m\]"; LIGHT_GRAY="\[\033[0;37m\]"; YELLOW="\[\033[1;33m\]";
    function parse_git_branch {
        git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ \(\1\)/';
    }
    function parse_git_status {
        git status 2> /dev/null | sed -e '/(working directory clean)$/!d' | wc -l;
    }
    function check_git_changes {
        # tput setaf 1 = RED, tput setaf 2 = GREEN
        [ `parse_git_status` -ne 1 ] && tput setaf 1 || tput setaf 2
    }
    export PS1="($(whoami)@$(hostname)) \[$(tput bold)\]\[$(tput setaf 2)\]$(date +%F\|%R) $YELLOW\w\[\$(check_git_changes)\]\$(parse_git_branch)$LIGHT_GRAY\n $ "
    export PS4='+(${BASH_SOURCE}:${LINENO})'
    ;;
*)
    ;;
esac

set -o emacs
umask 002
export HISTTIMEFORMAT="%F %T "
shopt -s histappend

export TZ='/usr/share/zoneinfo/US/Eastern'
export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"

alias l='ls -lF'
alias ll='ls -alF'

if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre

Export Gimp layers as separate files

Install plugin at https://github.com/khalim19/gimp-plugin-export-layers

Extract the export_layers.py file and the export_layers directory to [home directory]/.gimp-2.8/plug-ins.

If necessary, make the export_layers.py file executable, e.g. from the terminal:

chmod +x "export_layers.py"

Collate PDF documents

pdftk $(ls file*.pdf|sort) cat output combined.pdf