VirtualBox Ubuntu xterm Guest <-> Host copy & paste

When copying and pasting between guest VB Ubuntu huest and my host Windows machine Xterm does not work well (other applications have no problem).
Instead, I choose Gnome terminal (/usr/bin/gnome-terminal), which I like more due to smooth Copy&Paste, clear deafult colors scheme and better integration with Ubuntu desktop ("New terminal" menu on docked icon instead of "mid-click" - go figure that out - way of opening a new window with Xterm).

Manually install Oracle Java SDK on Ubuntu

The tar.gz provided by Oracle don't have an actual installation process. You just extract those files to a location you want and add them to your path. So the process is the following:

• Make a folder for your Java installs, e.g.
  

  sudo mkdir /usr/local/java/

• Download & extarct there a .tar.gz from Oracle (here I will be using jdk-8u60-linux-x64.tar.gz);
• Create a file /etc/profile.d/oraclejdk.sh with the following content (adapt the paths to reflect the path where you stored your JDK):
  

  export J2SDKDIR=/usr/local/java/jdk1.8.0_60/
  export J2REDIR=/usr/local/java/jdk1.8.0_60/jre
  export PATH=$PATH:/usr/local/java/jdk1.8.0_60/bin:/usr/local/java/jdk1.8.0_60/db/bin:/usr/local/java/jdk1.8.0_60/jre/bin
  export JAVA_HOME=/usr/local/java/jdk1.8.0_60
  export DERBY_HOME=/usr/local/java/jdk1.8.0_60/db

Done! Those paths will only be recognized after you logout or restart, so if you want to use them right away run source /etc/profile.d/oraclejdk.sh.

(slightly modified source)

Trying AKKA, Java, Scala, IntelliJ, Aerospike, Maven & Git with BitBucket on Windows

Discalimer: it is not a clean "how-to guide" for any of teh above tools or technologies. It's more like a record of my backs and forwards exploring this route, a list of thoughts, foundings and decisions I made while trying to make the things work for me.

I wanted to try some AKKA / Scala code which is new for me, and thought of creating a sample code which puts smth to / gets something from Aerospike DB. My primary machine is Windows 8.1 and I run VirtualBox with a varity of Linux distributions.

Reading the docs I understood that installing Aerospike on Windows actually does not make sense since there is no native Windows code for Aerospike. Intsead, the DB comes with Vagrant VM and it should run inside VirtualBox anyway, so I simply installed Aerospike on aVirtualBox Ubuntu VM following these simple steps: http://www.aerospike.com/docs/operations/install/linux/ubuntu/.

It worked like a charm from the first try, so I went to the next step: AKKA/Scala/Java and IntelliJ. I installed the latest IntelliJ Community (v15 Preview) https://www.jetbrains.com/idea/download/
and Typesafe Activator at http://akka.io/downloads/.

Activator is a web server packed with documentation, explanations and samples, and it can generate ready-to-use Eclipse and IntelliJ projects for each sample, providing both Java and Scala code in every project. When you start the process via suplied "./activator.bat" file it opens a new browser tab showing the list of samples and other content. Quite sure inside it is build with Typesafe' Play framework. I found it a pretty nice thing to start with.

So I went to "Hello Akka" sample in my browser and generated an IntelliJ project, which I opened in the brand new IntelliJ. I wanted the project to be managed by Maven, so I needed to add the Aerospike dependencies to "pom.xml" Maven file. But there was no "pom.xml" in the IntelliJ project generated by Typesafe Activator. Right-click on the "hello-akka" project name -> "Add Framework Support" -> selecting "Maven" added "pom.xml" to my new project (source), so I could add there the dependency code

<dependencies>
  <dependency>
    <groupId>com.aerospike</groupId>
    <artifactId>aerospike-client</artifactId>
    <version>3.1.5</version>
  </dependency>
</dependencies>

as per the Aerospike documentation. Aerospike currently does not have a Scala client library, so the initial plan of Scala tests did not work - I have to write the client code in Java.

If you do not have a working stand-alone Maven on your computer download and install it, so you can build Aerospike examples either with "mvn package" or with the provided "./build_all" script (seems the both commands do exactly the same) right from the command line. You will also need a working Git install for your Windows computer.

Default Maven settings location on Windows 8+ is "C:\Users\\.m2". The remote repositories are specified in "settings.xml" file (details). Your $MAVEN_HOME/conf folder contains a default "settings.xml" file which you can copy in your "Users" folder if something went wrong in the user' copy.

In order to work with a remote Git repository such as GitHub or Bitbucket you will need SSH access to them, otherwise you will be prompted for a password on each Git push. You should have a SSH key for that. This manual explains how to generate / find your SSH keys for GitHub, this one is the similar explanation for BitBucket.
If you do not have a key or if you need yet another SSH key (multiple Bitbuckets accounts require multiple SSH keys, one per each account) you can run this command in Git Bash to generate the new key:

ssh-keygen -t rsa -f ~/.ssh/ -C "my@email.com"

After creating the SSH key open Git Bash and create a ~/.bashrc file with the following content:

SSH_ENV=$HOME/.ssh/environment
 
# start the ssh-agent
function start_agent {
    echo "Initializing new SSH agent..."
    # spawn ssh-agent
    /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
    echo succeeded
    chmod 600 "${SSH_ENV}"
    . "${SSH_ENV}" > /dev/null
    /usr/bin/ssh-add
}
 
if [ -f "${SSH_ENV}" ]; then
     . "${SSH_ENV}" > /dev/null
     ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        start_agent;
    }
else
    start_agent;
fi

Next you can add print & copy the content of SSH key file

cat ~/.ssh/

and paste it to BitBucket or GitHub web UI for your account or your team. Also note this Bitbucket article for further help on multiple accounts / SSH keys.

For multiple BitBucket  accounts we need to create a ~/.ssh/config file with smth like that (note the indent, it's required):

Host personal
 User me
 Hostname bitbucket.org
 PreferredAuthentications publickey
 IdentitiesOnly yes
 IdentityFile ~/.ssh/id_personal
Host work
 User work_user
 Hostname bitbucket.org
 PreferredAuthentications publickey
 IdentitiesOnly yes
 IdentityFile ~/.ssh/id_work

and your Git commands should address these new hosts instead of deafult 'bitbucket.org' or 'github.org':

git clone git@personal:myteam/repo.git

An aletrnative to the Git clone command above could be a Git fetch (the 2-nd answer here) as listed below:

git init
git remote add origin git@personal:myteam/repo.git
git fetch origin
git checkout -b master --track origin/master

Next I had to switch off my VBox Aerospike VM and add "Port Forwarding" (VBox ->select VM -> Settings -> Network -> select your current NW adaptor -> Advanced), forwarding my machine' port 3000 to the VM' port 3000 so I can access the DB from the code by pointing to "127.0.0.1" localhost.

Other suggestions how to access a remote service on a VirtualBox VM actually did not work for me, so ended up with teh Ports Forwarding for now. It's good to start the VM again and manually run Aerospike:

sudo service aerospike start

Now wait for a few seconds and try in the  XTerm in your VM the following:

telnet localhost 3000

if everything is OK you should connect to your local Aerospike DB. Next, open "cmd.exe" on your host machine (install Microsoft telnet utility for your Windows if you did not do it yet) and run:

telnet localhost 3000

on your Windows machine. Assuming your port forwarding was properly defined you should be now connected to Aerospike on your VirtualBox VM.

Now it's time to get back to IntelliJ, but first take a break and read "Java client best practices" before moving on with coding with Aerospike. The first try with Java Aerospike client looks this way

import com.aerospike.client.*;
import com.aerospike.client.policy.BatchPolicy;
import com.aerospike.client.policy.ClientPolicy;
import com.aerospike.client.policy.WritePolicy;

public class MyAerospikeTest {

    public static void main( String[] argv){
        Host[] hosts = new Host[] {
                new Host("127.0.0.1", 3000),
//                new Host("another.host", 3000),//                new Host("and.another.host", 3000)        };

        AerospikeClient client = new AerospikeClient( new ClientPolicy(), hosts);
        // write a new value following the example: http://www.aerospike.com/docs/client/java/usage/kvs/write.html
        // Initialize writePolicy.        WritePolicy writePolicy = new WritePolicy();
        writePolicy.timeout = 5;  // 5 millisecond timeout.
        // Write multiple values.        Key key = new Key( "test", "myset", "mykey");
        Bin bin1 = new Bin( "name", "John");
        Bin bin2 = new Bin( "age", 25);
        client.put( writePolicy, key, bin1, bin2);

        System.out.println( "Added a new record to Aerospike DB!");

        // Now let's see what we wrote there        BatchPolicy batchPolicy = new BatchPolicy();
        batchPolicy.timeout = 2; // 2 ms timeout on read        // Get the record        Record whatWeWrote = client.get( batchPolicy, key);
        System.out.println( "Got the record from DB");
        // get some bins from the record (note to get just two bins "name" and "age" we could use: Record record = client.get(policy, key, "name", "age"); )        System.out.println( "Name: " + whatWeWrote.bins.get("name") + ", Age: " + whatWeWrote.bins.get("age"));

        // delete the record        client.delete( writePolicy, key);
        System.out.println( "And now removed the record from DB");

        client.close();

        System.out.println( "Closed connection and exited. See you soon");
    }
}

and when it runs it produces the expected output:

Added a new record to Aerospike DB!
Got the record from DB
Name: John, Age: 25
And now removed the record from DB
Closed connection and exited. See you soon

Process finished with exit code 0

So far so good. Moving forward, I was quite surprised reading about configuration challenges associated with the Aerospike namespaces. Storage receipts added even more confusion: what storage configuration should be requested in my use cases and how I am supposed to know that? I definitely do not want to get into some namespace limit, neither in terms of defined space not for the access time / latency once the data set grows up. Meanwhile I left it for the future research - I still need to move on with putting all things to work together.

The good surprise was Secondary Indexes and the query language, including aql tool. Clustering and monitoring for Community Edition also look promising. And Java docs are very comprehensive.

And here is yet another surpirse: it appears that IntelliJ (including the Community Edition) can convert any Java file to Scala with "Add framework support - >Scala" switched on for a Project / Module (Refactor menu / CTRL+SHIFT + G). Meaning, you can take open-source Aerospike client and rebuild it in Scala, and then contrubute a Scala plugin back to Aerospike. Or you can write Java code around Aerospike Java library and convert only your code to Scala with IntelliJ. And of course you can simply write your Scala code down and call Aerospike Java client from there as well.

In IntelliJ, in order to let module A in a project to be aware of packages / classes of module B we to add "A on B dependency". For Maven project / modules it is easy - add smth like

<dependency>
    <groupId>my-group-id</groupId>
    <artifactId>my-artifact-id</artifactId>
    <version>1.0</version>
</dependency>

to "A" pom.xml file, where the above IDs are like they are specified for your module "B".

Aeropsike namespaces can only be added by adding the namespace definition to
 /etc/aerospike/aerospike.conf
file. For the possible configuration parameters please refer to http://www.aerospike.com/docs/reference/configuration/.  After the new namespace is configured restart the Aerospike server.
(also note this post to avoid issues when adding namespaces. In general, namespaces can be RAM (with or without HHD for persistence) and SSD / Flash.

In the same way, the only option to delete a namespace in Aerospike is stopping the service, deleting the namespace from the configuration file, removing the associated file and then restarting the server.

Quite a lot of what compared to DELETE DATABASE AAA
MySQL SQL statement.

I found this nice Scala (wrapper) library for Aerospike: https://github.com/Tapad/scaerospike
and the blog post Aerospike on scaling:
http://engineering.tapad.com/blog/2014/08/aerospike-at-tapad-how-we-scale/

More about scaling up: http://www.slideshare.net/AerospikeDB/linked-in-twitter-facebook-google-email-embed-configuring-aerospike-part-2
Slide 7: in the clustered environment adding / removing a namespace requires cluster-wide restart
Slide 8: keep nodes identical to run at full cluster' capacity
Slide 9, 24: like mentioned by Tapad guys in the above post high-watermark and default TTL look to be the most important namespace configuration parameters. TTL / Expiration logic explained at Slide 25. NOte that if a record is updated the TTL is restarted.
Slide 11: A single record always belong to a single node
Slide 28: At high watermark server drops items with closest TTL.
Slide 29: Aerospike can stop writing new records if "stop-writes" is reached.


For now I got everything up and running with Java and Aerospike' handy Java API. I still have to look into Tapad' open source Scala Aerospike client.

Allowing secondary monitor for VirtualBox (5.0.4) VM in three easy steps

1. Power off VM. Go to Settings->Display, change number of monitors to 2. Add video memory if there is a warning.
2. Run the VM. Open View->Virtual screen 2 menu and select "Enabled". The second VM window appears.
3. As per your VM OS go to Settings / Control Panel and modify Display so the OS can use both monitors, just like with the normal Windows or Linux computer. For Ubuntu it is Settings->Display, for Windows you can right-click on desktop background and select "Screen Resolution..." menu.

Installing Ubuntu 14.04 in VirtualBox for NGiNX modules development

For teh first time, after many years with RHEL / FEdora / CentOS family I decded to give a try to Ubuntu. Why? First, because it is the Linux of choice at AWS, after Amazon Linux of course.
Second, in order to learn something different. Third, in order to make some comparison by myself.

First surprise with Ubuntu 14.04.03 which I installed with Oracle VirtualBox (I gave this development VM 4 CPUs, 12 GB RAM, 40GB HD) is that there is no root access. I defined a user during the install and this user got 'sudo' permissions. So either I have to write 'sudo' every time or allow root access. The latest is not recommended, and this explains why the root access is by default disabled at Ubuntu. In general, I agree with the idea that there is no need to run any user process as a root, and for any system I develop I actually define a specific user which such a system uses. So let's see if I can manage it without enabling root account.

After installing a basic server configuration I wanted to add a Desktop so I could add development IDE there. The only thing I needed to run was:

sudo apt-get install ubuntu-desktop

(as per this link) followed by reboot did the job. Next, I added VBox Guest Additions and got a full-screen resizable Desktop. After that I started to look how I can open a terminal window :). This was a real challenge since "Search" for 'xterm' using top-left Ubuntu icon only gave me the reference documentation for that. It did not run the terminal process though.
CTR-ALT-T worked, but I could not find any other app as well, even not a web browser.

Since I do dnot save the snapshot for teh basic server install I decided to give up and re-installed Ubuntu 14.04 server. Once the basic server install was completed I ran

sudo apt-get --install-recommends install ubuntu-desktop

It took about 15 minutes on my i7 Windows 8.1 machine to install all the required packages, and than I got a black screen with no text. As per System Monitor the VirtualBox process was keep writing things on disk and kept using CPU. First time it happened I killed the process after 10 minutes or so and started over. Next time it happened again, so I decided to give it a chance and left it open.
At the end I found out that I just had to press ENTER and the text is now shown again!
sudo reboot now.
I started VirtualBox Guest Additions CDROM, than Rebooted the machine (had to repeat it twice until it worked) and got teh working Desktop environment.

All-in-all, about 4h to get there with trial and errors.

Next, I installet NetBeans 8.1beta for C++/Java/Scala development. It requested to install JDK (8u60). I also had to install GNU C++ compiler:

sudo apt-get install g++

and CMake

sudo apt-get install cmake

I also had to install PCRE, zlib and a few more librbaries on Ubuntu in order to properly build quite a minimal configuration of NGiNX from the sources:
sudo apt-get install libpcre3 libpcre3-dev zlib1g-dev build-essential libssl-dev

Now I configured NGiNX from the source to be installed in a development path:
cd ; ./configure --prefix=/home/david/dev/nginx
It created Makefile in that folder. I started NetBeans with "New C/C++ project from existing source", pointed to that  and got the Netbeans project for NGiNX.

I had to modify Netbeans project (Right-click on project -> Properties)
adding
${MAKE} -f Makefile install 
instead of 
${MAKE} -f Makefile
to Build section,
and adding 
./configure --prefix=/home/david/dev/nginx
to Pre-build section (check "Pre-build" checkbox!)
After all, I got an IDE-compiled running NGiNX.

Installing s3cmd from the sources

1. Go to s3cmd GitHub home page: https://github.com/s3tools/s3cmd

2.  Install python, wget and unzip if not installed
yum install unzip python-pip wget

3. Download the archive with the sources:
wget https://github.com/s3tools/s3cmd/archive/master.zip

4. unzip, change directory and run install:

unzip master.zip

cd s3cmd-master

python setup.py install

 5. Run s3cmd and save configuration in the config file:
s3cmd --configure

(provide your access and secret keys on the prompt)

Create a new Git repository for new / existing project in Netbeans while putting it on Bitbucket

First, there are free accounts at Bitbucket.org, up to 10 users, no obligations, private repositories.

Create a Bitbucket account. Login. Go to "Repositories" menu at the top screen bar and select "Create repository". Add the repository name and other info and create the repository. When done, copy & paste the URL which now appears in the browser, e.g. "https://bitbucket.org/david_mail/my_new_repository".

Open your Netbeans project (I am currently using NB 8.0.2), right click on the project name, select "Versioning"->"Initialize Git Repository". Next, right-click again on the project name and select "Git"->"Commit".

Your project is no in th elocal repository. Let's connect it to teh remote repository so they'll auto-merge. At top NB menu open "Team"->"Remote"->"Push".
If it is the first time you add this Git repository put the Bitbucket' repository URL in "Specify Git Repository Location / origin"->"Repository URL" and provide your user credentials for Bitbucket. Hit "Next" and "Finish" buttons and you are done.