Wikimedia dumps

Using the Docker registry REST interface

Docker registry is a server application that stores and distributes Docker images. This note explain how to use curl, and optionally jq, to query the registry for information about container images and their history.

The server's endpoint is assumed to be in a shell variable, such as:
MY_REGISTRY=http://some.host:5000,

To list repositories:

curl https://MY.REGISTRY/v2/_catalog
...

Note: local development registries often do not have proper TLS certificates. For trusted sources and assuming the associated risks, one can bypass certificates and use an insecure connection by adding the curl flag -k/--insecure to the commands shown below.

To list available tags for a given image (e.g. myteam/myapp):
 
curl https://MY.REGISTRY/v2/myrepo/myapp/tags/list

To retrieve the manifest of a given image: (e.g. myteam/myapp):

curl https://MY.REGISTRY/v2/myrepo/myapp/manifests/latest

A common application of manifest queries is to catalog images and use lineage and other metadata on those images for automation. Such metadata is found in image manifests, which are hard to inspect visually due to their complexity. Using a JSON parser like jq can be used to produce a readable report.

Image registry summary report

The following script will produce a simple summary listing every tag for a given image along with SHA identifier and creation date. Temporary files are used to cache intermediate results.

#=== Configure repo and image
REPO_URL=https://my.registry
IMAGE_REPO=myrepo/myimage
BASE_URL=$REPO_URL/v2/$IMAGE_REPO

#=== Download tags

curl -ks $BASE_URL/tags/list > /tmp/tags
#=== Download each tag's manifest, all concatenated
for tag in $(jq -r .tags[] /tmp/tags)
do
  curl -ks $BASE_URL/manifests/$tag
done > /tmp/manifests
#=== Parse manifests and print metadata
cat /tmp/manifests |
  jq -r '{tag:.tag, info:.history[0].v1Compatibility |
    fromjson |
    {created:.created, sha:.config.Image}}' |
  tee /tmp/repo.data.json

 The output can be sorted by timestamp thus:

cat /tmp/repo.data.json | jq -s 'sort_by(.info.created)' 

List all Firefox bookmarks

Firefox stores bookmark data in profile folders. For Linux, this is typically ~/.mozilla/firefox (Windows: %APPDATA%\Mozilla, Mac: ~/Library/Mozilla). Each profile has a dedicated directory, e.g. mwad0hks.default, under which a file named places.sqlite holds a database containing bookmarks. To retrieve every bookmark in the profile:

cd ~/.mozilla/firefox/<profile>.default # replace <profile>

sqlite3 places.sqlite # open prompt in sqlite client
select * from moz_places; -- list bookmarks

A simple Elasticsearch index

Create and populate a simple Elasticsearch index

curl -XPUT http://dev01:9200/test \
   -H 'Content-type:application/json' -d'{"
      settings":  {
         "number_of_shards":1,
         "number_of_replicas":1
      },
      "mappings": {
         "type": {
            "properties": {
               "f1": {
                  "type":"string"
               },
               "f2": {
                  "type":"string"
               },
               "f3": {
                  "type":"string"
               }
            }
          }
        }
      }'

Kubernetes development with minikube

Minikube's default memory parameters are fine for running small stacks, but more complex deployments, such as those including telemetry components like Prometheus or ELK, will demand larger resource allocations.

$ minikube delete
$ minikube start

Show Kubernetes dashboard on minikube:

$ minikube dashboard --url

without --url, the command above command launches a local browser.

List services in minikube:

$ minikube service list
|-------------|----------------------|-----------------------------|
|  NAMESPACE  |         NAME         |             URL             |
|-------------|----------------------|-----------------------------|
| default     | kubernetes           | No node port                |
| kube-system | kube-dns             | No node port                |
| kube-system | kubernetes-dashboard | http://192.168.99.102:30000 |
|-------------|----------------------|-----------------------------|
Minikube ports are bound to local IP addresses exposed by VM virtual net adapters. When accessing the minkube host remotely, you can either use a SSH tunnel, or use NAT in your VM runtime. For VirtualBox, e.g.:

$ vboxmanage controlvm "minikube" natpf1 "k8s,tcp,,30000,,30000"

To inspect the resulting change, check the NAT rules for the NIC:

$ vboxmanage showvminfo minikube
[...]
NIC 1:           MAC: 080027D238E5, Attachment: NAT, Cable connected: on, Trace: off [...]
[...]
NIC 1 Rule(2):   name = k8s, protocol = tcp, host ip = , host port = 30000, guest ip = , guest port = 30000
NIC 1 Rule(3):   name = ssh, protocol = tcp, host ip = 127.0.0.1, host port = 35283, guest ip = , guest port = 22

Set/unset environment variables from the JVM (Groovy)

Dirty hack recommended only for testing. Only ever tested in Linux; may not work at all on Windows.
 
def setEnv(String key, String value) {
    try {
        // Dirty hack to set environment variables from JVM
        // Only tested in Linux - may not work in Windows at all
        Map<String, String> env = System.getenv()
        Class<?> cl = env.getClass()
        Field field = cl.getDeclaredField("m")
        field.setAccessible(true)
        Map<String, String> writableEnv = (Map<String, String>) field.get(env)
        writableEnv.put(key, value)
    } catch (Exception e) {
        throw new IllegalStateException("Failed to set environment variable", e)
    }
}

def unsetEnv(String key) {
    try {
        // Dirty hack to unset environment variables from JVM
        // Only tested in Linux - may not work in Windows at all
        Map<String, String> env = System.getenv()
        Class<?> cl = env.getClass()
        Field field = cl.getDeclaredField("m")
        field.setAccessible(true)
        Map<String, String> writableEnv = (Map<String, String>) field.get(env)
        writableEnv.remove(key)
    } catch (Exception e) {
        throw new IllegalStateException("Failed to set environment variable", e)
    }
}

Build emacs from source (CentOS/RHEL 7)

I often have development systems where I need my favorite editor: Emacs Live. Since Emacs Live requires Emacs 24.4+, and as of writing the latest version in the standard yum repositories is 24.3, rather than fish around for a potentially dangerous RPM, I strongly prefer to build from source.

Most of my remote development hosts do not typically have a build chain installed, so to set one up:

sudo yum group install "Development Tools"

Download and extract the intended Emacs sources, e.g.:

wget http://mirror.clarkson.edu/gnu/emacs/emacs-25.3.tar.gz
tar xvf emacs-25.3.tar.gz

Configure the build:

cd emacs-25.3

./configure
# use --with-x=no with configure to exclude the X11 UI from the build, which is ideal for text-only remote systems

I often, however, run into the following:

...
configure: error: The required function 'tputs' was not found in any library.
The following libraries were tried (in order):
  libtinfo, libncurses, libterminfo, libtermcap, libcurses
Please try installing whichever of these libraries is most appropriate
for your system, together with its header files.

For example, a libncurses-dev(el) or similar package.

Which can be fixed by installing ncurses:

sudo yum install ncurses-devel

Retry the configure step:

./configure --with-x=no

And proceed to build:

make
sudo make install

Finally, launch (with high color terminal):

export TERM=xterm-256color && emacs