Set/unset environment variables from the JVM (Groovy)

Dirty hack recommended only for testing. Only ever tested in Linux; may not work at all on Windows.
 
def setEnv(String key, String value) {
    try {
        // Dirty hack to set environment variables from JVM
        // Only tested in Linux - may not work in Windows at all
        Map<String, String> env = System.getenv()
        Class<?> cl = env.getClass()
        Field field = cl.getDeclaredField("m")
        field.setAccessible(true)
        Map<String, String> writableEnv = (Map<String, String>) field.get(env)
        writableEnv.put(key, value)
    } catch (Exception e) {
        throw new IllegalStateException("Failed to set environment variable", e)
    }
}

def unsetEnv(String key) {
    try {
        // Dirty hack to unset environment variables from JVM
        // Only tested in Linux - may not work in Windows at all
        Map<String, String> env = System.getenv()
        Class<?> cl = env.getClass()
        Field field = cl.getDeclaredField("m")
        field.setAccessible(true)
        Map<String, String> writableEnv = (Map<String, String>) field.get(env)
        writableEnv.remove(key)
    } catch (Exception e) {
        throw new IllegalStateException("Failed to set environment variable", e)
    }
}

Build emacs from source (CentOS/RHEL 7)

I often have development systems where I need my favorite editor: Emacs Live. Since Emacs Live requires Emacs 24.4+, and as of writing the latest version in the standard yum repositories is 24.3, rather than fish around for a potentially dangerous RPM, I strongly prefer to build from source.

Most of my remote development hosts do not typically have a build chain installed, so to set one up:

sudo yum group install "Development Tools"

Download and extract the intended Emacs sources, e.g.:

wget http://mirror.clarkson.edu/gnu/emacs/emacs-25.3.tar.gz
tar xvf emacs-25.3.tar.gz

Configure the build:

cd emacs-25.3

./configure
# use --with-x=no with configure to exclude the X11 UI from the build, which is ideal for text-only remote systems

I often, however, run into the following:

...
configure: error: The required function 'tputs' was not found in any library.
The following libraries were tried (in order):
  libtinfo, libncurses, libterminfo, libtermcap, libcurses
Please try installing whichever of these libraries is most appropriate
for your system, together with its header files.

For example, a libncurses-dev(el) or similar package.

Which can be fixed by installing ncurses:

sudo yum install ncurses-devel

Retry the configure step:

./configure --with-x=no

And proceed to build:

make
sudo make install

Finally, launch (with high color terminal):

export TERM=xterm-256color && emacs

Linux audio

killall jackdbus

Until ps -ef | grep jack shows no jack processes
pulseaudio --kill

aplay -l

aplay -D hdmi alert.wav

alsamixer 

Teradata tricks

select * from dbc.SessionInfo;

SELECT   SYSLIB. AbortSessions(1 ,'dbc', 0,'Y' ,'Y') ; -- All sessions for a user

SELECT   SYSLIB. AbortSessions(1 ,'DBC', 123340,'Y' ,'Y') ; -- Specific session

Param1: Hostno

Param2:UserName

Param3: SessionNo

Param4: LogoffSessions

Param5: UserOverride

SELECT * FROM TABLE (MonitorSession (1, '*',0 )) AS dt WHERE PEstate <> 'Idle' or AMPstate <> 'Idle' ;

---

SELECT      TOP 10 username , clientAddr, defaultDatabase , CacheFlag ,

collectTimeStamp ,

StartTime, ElapsedTime , NumResultRows, AMPCPUTime, ERRORCODE ,

StatementType , QueryText , NumOfActiveAmps, SpoolUsage , ReqIoKB ,

ReqPhysIO

FROM dbc. qrylog

WHERE     defaultDatabase LIKE 'TPCDS%'

ORDER    BY StartTime DESC;

BEGIN QUERY LOGGING

WITH      ALL LIMIT SQLTEXT= 0

    ON   ALL;

SELECT TOP 10 *

FROM dbc. QryLogTDWM

ORDER BY collectTimeStamp DESC

SELECT TOP 100 *

FROM dbc. QryLogV

WHERE StatementType IN ('Insert' ,'Update', 'Delete')

ORDER BY collectTimeStamp DESC

SELECT TOP 100 *

FROM dbc. QryLogEvents

ORDER BY collectTimeStamp DESC

---

$ sudo pdestate -a
root's password:
PDE state: DOWN/TDMAINT

$ sudo /etc/init.d/tpa start
Teradata Database Initiator service is starting...
Teradata Database Initiator service started successfully.

$ sudo pdestate -a
PDE state is RUN/STARTED.
DBS state is 1/3: DBS Startup - Starting AMP Partitions

$ sudo pdestate -a
PDE state is RUN/STARTED.
DBS state is 4: Logons are enabled - Users are logged on

---

COLLECT STATISTICS ON tablename COLUMN columnname;

COLLECT STATISTICS ON tablename INDEX ( columnname);

COLLECT STATISTICS ON tablename INDEX ( col1, col2 , ...);

HELP STATISTICS tablename;

COLLECT STATISTICS tablename; -- refresh table statistics

DROP STATISTICS ON tablename ;

DIAGNOSTIC HELPSTATS ON FOR SESSION;

Then run query EXPLAIN. Optimizer will return something like the following at the end of the query plan:

BEGIN RECOMMENDED STATS FOR FINAL PLAN->

     -- "COLLECT STATISTICS COLUMN (I_ITEM_ID) ON TPCDS1000G.item"

     (High Confidence)

     -- "COLLECT STATISTICS COLUMN (D_DATE) ON TPCDS1000G.date_dim"

     (High Confidence)

     -- "COLLECT STATISTICS COLUMN (S_STORE_ID) ON TPCDS1000G.store"

     (High Confidence)

     TPCDS1000G.store_sales" (High Confidence)

     -- "COLLECT STATISTICS COLUMN (P_PROMO_SK) ON TPCDS1000G.promotion"

     (High Confidence)

     -- "COLLECT STATISTICS COLUMN (D_DATE_SK) ON TPCDS1000G.date_dim"

     (High Confidence)

     <- END RECOMMENDED STATS FOR FINAL PLAN

     BEGIN RECOMMENDED STATS FOR OTHER PLANS ->

     -- "COLLECT STATISTICS COLUMN (PARTITION) ON

     TPCDS1000G.store_sales" (High Confidence)

     <- END RECOMMENDED STATS FOR OTHER PLANS

---

Collect Full Statistics

• Non-indexed columns used in predicates
• All NUSIs with an uneven distribution of values * 
• NUSIs used in join steps
• USIs/UPIs if used in non-equality predicates (range constraints)
• Most NUPIs (see below for a fuller discussion of NUPI statistic collection)
• Full statistics always need to be collected on relevant columns and indexes on small tables (less than 100 rows per AMP)

Can Rely on Random AMP Sampling   

• USIs or UPIs if only used with equality predicates
• NUSIs with an even distribution of values
• NUPIs that display even distribution, and if used for joining, conform to assumed uniqueness (see Point #2 under “Other Considerations” below)
• See “Other Considerations” for additional points related to random AMP sampling

Option to use USING SAMPLE

• Unique index columns
• Nearly-unique columns or indexes**

Collect Multicolumn Statistics

• Groups of columns that often appear together in conditions with equality predicates, if the first 16 bytes of the concatenated column values are sufficiently distinct. These statistics will be used for single-table estimates.
• Groups of columns used for joins or aggregations, where there is either a dependency or some degree of correlation among them.***  With no multicolumn statistics collected, the optimizer assumes complete independence among the column values. The more that the combination of actual values are correlated, the greater the value of collecting multicolumn statistics will be.

Basic HBase in Clojure

(println (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))) ;; get class path in REPL

# in REPL

(require '[clojure-hbase.core :as hbase])

(import [org.apache.hadoop.hbase HBaseConfiguration HConstants KeyValue])

(import [org.apache.hadoop.hbase.client HTablePool Get Put Delete Scan Result RowLock HTableInterface])

(hbase/set-config (hbase/make-config {

     :zookeeper.znode.parent "/hbase-unsecure" 

     :hbase.zookeeper.property.clientPort "2181"

     :hbase.cluster.distributed "true"

     :hbase.zookeeper.quorum "hdp005-3,hdp005-21,hdp005-23"

}

))

(hbase/table "tweets-test")

Clojure REPL and tricks

lein repl :start :port 10010

(use 'clojure.repl 'clojure.pprint)

(setq nrepl-popup-stacktraces-in-repl nil)
(setq nrepl-auto-select-error-buffer nil)

Zoom

You can use `C-x C-+’ and ‘C-x C--’ (‘text-scale-adjust’) to increase or decrease the buffer text size (`C-+’ or ‘C--’ to repeat).

Very important: add cider plugin in project.clj

(defproject lucy "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [
                 [org.clojure/clojure "1.6.0"]
                 [clucy "0.4.0"]
                 ]
  :plugins [[cider/cider-nrepl "0.6.0"]]
  )

In CIDER

C-c M-n: Set REPL namespace from buffer

C-x C-e: Eval preceding line

C-c C-k: compile buffer

C-x C-c: eval region

C-c C-c: abort eval

M-: Read a single Emacs Lisp expression in the minibuffer, evaluate it, and print the value in the echo area (eval-expression).

C-x h selects the entire buffer.

C-M-\ reindents the selected region.

C-M-@ select s-expression

M-. Jump to the definition of a symbol. If invoked with a prefix argument, or no symbol is found at point, prompt for a symbol.

M-@ (mark-word) puts the mark at the end of the next word

Use M-< to move to the beginning of the buffer, and M-> to move to the end

C-v to scroll down, and M-v to scroll up

M-d Kill up to the end of a word (kill-word).

M-\ Delete spaces and tabs around point (delete-horizontal-space)

M-<SPC> Delete spaces and tabs around point, leaving one space (just-one-space)

M-^ Join two lines by deleting the intervening newline, along with any indentation following it (delete-indentation)

C-t Transpose two characters (transpose-chars).

M-t Transpose two words (transpose-words).

C-M-t Transpose two balanced expressions (transpose-sexps).

C-x C-t Transpose two lines (transpose-lines).

M-x auto-revert-tail-mode (tail a file)

You can hide the *nrepl-connection* and *nrepl-server* buffers from appearing in some buffer switching commands like switch-to-buffer(C-x b) like this:
(setq nrepl-hide-special-buffers t)

---

M-% string <RET> newstring <RET>

Replace some occurrences of string with newstring.

C-M-% regexp <RET> newstring <RET>

Replace some matches for regexp with newstring.

---

<SPC>

to replace the occurrence with newstring.

<DEL>

to skip to the next occurrence without replacing this one.

, (Comma)

to replace this occurrence and display the result. You are then asked for another input character to say what to do next. Since the replacement has already been made, <DEL> and <SPC> are equivalent in this situation; both move to the next occurrence.You can type C-r at this point (see below) to alter the replaced text. You can also type C-x u to undo the replacement; this exits the query-replace, so if you want to do further replacement you must use C-x <ESC> <ESC> <RET> to restart (see Repetition).

<RET>

to exit without doing any more replacements.

. (Period)

to replace this occurrence and then exit without searching for more occurrences.

!

to replace all remaining occurrences without asking again.

Y (Upper-case)

to replace all remaining occurrences in all remaining buffers in multi-buffer replacements (like the Dired `Q' command which performs query replace on selected files). It answers this question and all subsequent questions in the series with "yes", without further user interaction.

N (Upper-case)

to skip to the next buffer in multi-buffer replacements without replacing remaining occurrences in the current buffer. It answers this question "no", gives up on the questions for the current buffer, and continues to the next buffer in the sequence.

^

to go back to the position of the previous occurrence (or what used to be an occurrence), in case you changed it by mistake or want to reexamine it.

---

Print object members

(doseq [m (.getMethods (type index))] (println m))

(use 'clojure.reflect 'clojure.pprint)

(pprint (reflect "hello"))

{:bases
 #{java.io.Serializable java.lang.Comparable java.lang.Object
   java.lang.CharSequence},
 :flags #{:public :final},
 :members
 #{{:name valueOf,
    :return-type java.lang.String,
    :declaring-class java.lang.String,...

List classpath

(defn classpath []

(seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))))

Expand macros

(macroexpand '(time (print "timing")))

;; or better yet

(clojure.pprint/pprint (macroexpand '(time 1)))

List members in namespace

(require '[clojure-hbase.core :as hbase])

(dir clojure-hbase.core)

Show function source

user=> (source hbase/table)
(defn table
  "Gets an HTable from the open HTablePool by name."
  [table-name]
  (io!
   (.getTable (htable-pool) (to-bytes table-name))))

Find elements by regex match in collection

(filter #(re-find #"zoo" (key %)) (seq (hbase/make-config nil)))

clojure.core/all-ns
  Returns a sequence of all namespaces.

List files

(take 10 (file-seq (clojure.java.io/file ".")))

Basic Zookeeper in Clojure

Add [zookeeper-clj "0.9.1"] to dependencies

lein repl

(require '[zookeeper :as zk])

(.getChildren client "/" false) ;; watcher=false

#<ArrayList [hbase-unsecure, storm, zookeeper]>

(def ZK_HOSTS "127.0.0.1:2181,node1,node2,node3)

(def ZK_ROOT "/twitter-demo")(def client (zk/connect ZOOKEEPER_HOSTS :watcher (fn [event] (println event))))

(zk/create client ZK_ROOT :persistent? false)

(def version (:version (zk/exists client ZK_ROOT)))

(zk/create client ZK_ROOT :data (.getBytes "123143kdjkds") :persistent? false)

Dump Java class loaders including JAR physical paths

private static Iterator list(ClassLoader CL)

        throws NoSuchFieldException, SecurityException,

        IllegalArgumentException, IllegalAccessException {

    Class CL_class = CL.getClass();

    while (CL_class != java.lang.ClassLoader.class) {

        CL_class = CL_class.getSuperclass();

    }

    java.lang.reflect.Field ClassLoader_classes_field = CL_class

            .getDeclaredField("classes");

    ClassLoader_classes_field.setAccessible(true);

    Vector classes = (Vector) ClassLoader_classes_field.get(CL);

    return classes.iterator();

}

 

private void dumpClasses() {

    ClassLoader myCL = Thread.currentThread().getContextClassLoader();

    while (myCL != null) {

        log.warn("===> ClassLoader: " + myCL);

        try {

            for (Iterator iter = list(myCL); iter.hasNext();) {

                Class clazz = (Class) iter.next();

                log.warn("======> " + clazz.getName() + " @ "

                         + clazz.getProtectionDomain().getCodeSource().getLocation());

            }

        } catch (NoSuchFieldException e) {

            e.printStackTrace();

        } catch (IllegalAccessException e) {

            e.printStackTrace();

        }

        myCL = myCL.getParent();

    }

}

Magit

kudu: magit-20170913.1424

JACK Audio/ALSA on Ubuntu

Most Linux distributions (except audio-oriented distros such as Ubuntu Studio) have active memory limits too restrictive to operate JACK.

First, try ulimit -l. If the output is not 'unlimited' you will need to make it so by changing some configurations.

1. Your login user must be part of the groups 'audio' and 'realtime'. Typically:

groupadd realtime
usermod -a -G audio <my user>
usermod -a -G realtime <my user>

2. Increase memory limits and I/O priority for the audio group. Edit /etc/security/limits.d/audio.conf and make sure it contains:

@audio   -  rtprio     95
@audio   -  memlock    unlimited

3. Give the JACK daemon high priority:

dpkg-reconfigure -p high jackd

4. Enable PAM limits. Edit /etc/pam.d/common-session and add:

session required pam_limits.so

5. Log out and log back in to activate the changes

6. Start JACK. Make sure your output device is correctly referenced by hw:<n>.

jackd -R -d alsa -d hw:1 -r 44100

To list local devices:

aplay -l

Verify configuration in your software of choice, e.g. Ardour, LMMS, Audacity.

Using Docker behind a proxy

Proxies directly affect three independent aspects of Docker's operation: 

1. Docker daemon: for interacting with public repositories (e.g. `docker pull`)
2. Image building: building images via the Docker client almost always requires remote artifacts (e.g. package managers, build tools, curl)
3. Container runtime: running containers requiring internet access

Each of these aspects requires particular measures to operate correctly behind a proxy. The following sections explain configuration procedures for each of them.

Docker daemon

Docker daemon requires network access for pulling and pushing images from/to public repositories. To operate behind a proxy, the daemon must receive proxy settings as described below (applies to CentOS and related distributions -- for others, see applicable documentation).

Despite some claims in StackOverflow and other sites, Docker does not accept proxy configs in `/etc/docker/daemon.json` (https://docs.docker.com/engine/admin/systemd/#httphttps-proxy). Proxy variables must be set in the systemd startup configs for the service.

To update the daemon's proxy configuration, edit/create `proxy.conf` with your proxy settings, reload the configuration and restart the daemon:

sudo mkdir /etc/systemd/system/docker.service.d # not present by default

cat <<EOF > /etc/systemd/system/docker.service.d/proxy.conf
[Service]
Environment="HTTP_PROXY=http://your.proxy:3128/" "HTTPS_PROXY=https://your.proxy:3128/" "NO_PROXY=localhost,127.0.0.1,docker-registry.yourdomain.net"
EOF

sudo systemctl daemon-reload && sudo systemctl restart docker

The changes above will allow the daemon to access public registries to perform pull operations to retrieve images.

Image building

Image builds specified in Dockerfiles virtually always contain access operations on public resources, be it through curl, yum, npm or similar tools. For these operations to work behind a proxy, the container's build context must provide correct proxy settings.

The simplest but least flexible option is to hardcode proxy settings in your Dockerfile:

ENV http_proxy http://<PROXY_HOST>:<PORT>
ENV https_proxy https://<PROXY_HOST>:<PORT>

Notice both HTTP and HTTPS values are specified to ensure both protocols are configured. Replace <PROXY_HOST> and <PORT> with your network's proxy settings, e.g. `https://proxy.my-domain:3128`.

Many Linux applications will honor the `http(s?)_proxy` variable (e.g. curl, wget, many http client libraries for Python, Ruby and other scripted languages). There are, however, exceptions and irregularities in this area. See the documentation for the particular tool you intend to use. The CentOS package manager, yum, for example, requires explicit configuration via `/etc/yum.conf`. To allow yum to access packages during image building, include the following entries in your Dockerfile before the first invocation of yum:

RUN echo "http_proxy=http://proxy.my-domain:3128" >> /etc/yum.conf
RUN echo "https_proxy=https://proxy.my-domain:3128" >> /etc/yum.conf
# Make sure YUM proxy is set up first
RUN yum update -y && yum install curl

 

Plotting 3D sets in Jupyter

%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
import numpy as np

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
S = 50

def f(x,y):
    return np.exp(- x**2 - y**2)

def plot(x, y, f):
    xx, yy = np.meshgrid(X,Y)
    Z = map(lambda x: map(lambda y: f(x,y), Y), X)
    p = ax.plot_surface(xx, yy, Z, rstride=1, cstride=1, cmap=cm.jet,
                        linewidth=0.01, antialiased=True, shade=True)

X = np.linspace(-2,2,S)
Y = np.linspace(-2,2,S)
plot(X, Y, f)

Orbiter 2016 - Delta Glider IV autopilot

ALT+1  PRO400SPEC01  Taxiing hold speed (maintain a speed of 10 m/s)
ALT+2  PRO400SPEC18  Aproach hold speed (maintain a speed of 180 m/s)
ALT+3  PRO400SPEC25  Flight hold speed (maintain a speed of 250 m/s)
ALT+4  PRO300SPEC0   Docking auto (Automatically dock your DGIV)
ALT+5  PRO110SPEC0   Atmospheric flight autopilot (heading, altitude &speed)
ALT+6  PRO903SPEC40  Earth ascent to low orbit. (normal take-off)
ALT+7  PR105SPEC40   Automatic reentry

PRO104SPEC40 Manual rentry (Hold attitude with selected AOA)
PRO105SPEC40 Auto reentry (keep a low temperature reentry profile)
PRO200SPEC7  Manual Hover
PRO200SPEC8  Auto Hover
PRO500SPEC0  Null relative speed in regard of a close object (in space only)
PRO904SPECnn Earth ascent autopilot with automatic hover take-off.
PRO905SPECnn Moon ascent autopilot with automatic hover take-off.
PRO906SPECnn Mars ascent autopilot with automatic hover take-off.

Launch Azimuth = arcsin (cos (desired_orbital_inclination) / cos (launch_ latitude))

KSC(28.591) to ISS (51.56) => 45.075°(PRO904SPEC45)

Merci beaucoup Dansteph.

Numpad thrusters (translation mode):

• 0: hover +
• .: hover -
• 1: left
• 2: up
• 3: right
• 4: (inactive)
• 5: killrot
• 6: forward
• 7: (inactive)
• 8: down
• 9: retro