Pydoop Egg install on CentOS by Cloudera Hadoop Client

The Pydoop is a quick library for us to develop or research the Hadoop service and we have some prototype require using python to access the HDFS. Therefore Pydoop is the first choice to let script interact with HDFS files. However, the Pydoop installation require some hadoop library to make it compile-able on your dev machine. Hence, we choose the Cloudera hadoop client library to install on our dev machine for developing Pydoop script by eclipse.
First, you need to install the CDH4 Repository RPM to let your CentOS get the hadoop client software package. Then, you can yum the hadoop-client
# rpm -ivh cdh4-repository-1-0.noarch.rpm
# yum install hadoop-client

For pip installation, you need to assign the JAVA_HOME and HADOOP_HOME for python to compile the pydoop package.
# export JAVA_HOME=/urs/lib/jvm/java-1.6.0
# export HADOOP_HOME=/usr/lib/hadoop
# pip install pydoop==0.10.0

The High-Availability Design Paradigm of Application

We have often encounter the Design dilemma between Service Reliability and Application Development Cost. For High-Availability, we need put a lot effort aside from Business Requirement. However, too simple HA solution would have melt your business down when you facing the accident from all kind of Hardware Failure. Here we talk about three level's HA Paradigm from the perspective of implementation complexity.
1. Manual Switch: Service has basic Monitor Infrastructure to help you identify the Hardware failure and you can bring up the Application on the other spare capacity and keep the service continuity. Manual Switch is easily to adopt and take minimal cost. However, this is not even to be an HA Design. Because from the service interrupted to manual started, it might take over 30 minutes (Monitoring Interval usually take 5~10 minute to catch event and alert. Human check False Alarm. Confirmed and follow the SOP to start Application. Service resume.) This kind of low level HA is suitable for some none-timing critical mission such as file transfer, report generator etc. 30 minutes is tolerable for these kind of service flow.
2. Semi-Auto Switch: Some service might have strictly data consistency and require really rigid transaction result without race condition. At meanwhile, the interruption should not take over couple minutes. We usually design Active and Passive nodes and let those nodes coordinate with each other. So there will be only one Active node working at same time. Once the Active node shut down, the Passive one will start to take over the control and occupy a lock (usually in Database). Once the malfunctioned machine recovered, it will not proceed any transaction due to the lock has been acquired by the other partner. There are so many design like Database Cluster. The multiple nodes will take a vote under some quorum assignment and bring up another candidate as Active one for continuing the task. In a service flow, we will have an application behind the queue to maintain the data consistency and no service interruption before the queue. All the switch issue would be taken care after the queue. System interface seems ok. But the internal service flow would have couple minutes downtime and the queue would be a cushion to prevent the damage propagate to  other dependency system.
3. Active-Active Mode: This is most idealism for a service that every node has the same responsibility and no one's failure could make the service interrupted. However, sometimes this design might take a lot of over burden for all the application to communicate with each other for maintaining the data consistency or prevent from race condition issue during the transaction. Often, this kind of burden will drag down all the performance among whole cluster in poor design. Hence, only few scenario could adopt this feature without too much effort like Web Farm with only query capability (no transaction). Service flow that would only have one concurrent user connected to server at a time like ATM (you would only have on debit card, right?) For the application that focus on data availability, this design is pretty fascinated. But if your application is required to maintain the data consistency with multiple connection from  many concurrent users, the Active-Active Mode need a lot of time to enhance the performance, better Data structure for reducing the lock activities, crystal clear service flow and business purpose in case you need to expand the features in future business change.

Usually, we could take compromise for level 2 or 1. But if we could, why not Active-Active for completing your solution.

The way to figure out how your linux application installations and deployments are configured

There are several main distribution among Linux community and each of them have slightly different configuration upon the software package management and compiler prefix. Hence, this is crucial to learn what the current package deployed on your system:
1. Default Repository Manager: such as yum, apt-get, homebrew, you have to learn the main stream repository and deployment management system on you distribution.
2. Default Package Manager: such as rpm, dkpg, you need to learn the shell command about software installation shell command. And these command usually provide a database to record the relationship between package's dependencies. You might have a lot customized software package that provide by vendor which you can not found the open source software on public repository for you to yum or apt-get.
3. Useful shell command `locate`: we usually use this command to learn the installation made by ./configure and `make install`. This command is really useful when you have some cutting edge software that compiled and installed manually by your own. Usually, sometimes we use the command to check the library missing or any misconfigure package installation.
4. Finally, you have to check the following folder like /etc/init.d on CentOS for your command about `service` or `sbin/service` and `chkconfig`. These are the command for you to lookup the application would be started at booting.
5. If the file does exist but you still have trouble on link it, you could use ldd command to parse the ELF binary for checking all the dependencies' location such as
[user@server ~]$ ldd /usr/lib64/libtdsodbc.so.0
        linux-vdso.so.1 =>  (0x00007fffe3ea0000)
        libodbcinst.so.2 => not found
        libgnutls.so.26 => /usr/lib64/libgnutls.so.26 (0x00007f65b1a6a000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f65b1862000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f65b1645000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f65b12b0000)
        libtasn1.so.3 => /usr/lib64/libtasn1.so.3 (0x00007f65b10a0000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f65b0e8a000)
        libgcrypt.so.11 => /lib64/libgcrypt.so.11 (0x00007f65b0c14000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f65b1f71000)
        libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f65b0a10000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f65b080c000)

The Cent OS Workstation

Recently, I would like to use Eclipse to develop python Client for Hadoop usage such as HDFS and HBase. However, I found there is no way to build up a "pydoop" client on Windows Machine. Hence we have to use CentOS as our regular based workstation. Although I have the experience on Mac for daily usage (email, browser, documentation) but the stuff related to workstation are totally different from Sever Management and Office Work. The Distribution Config is Software Developement.
For better transition, I have to left some work still on windows workstation. Therefore, the interaction between new CentOS Workstation and original windows machine take its matter:
1. Install the RDP Client on CentOS
[root@new]# yum install xfreerdp
Then you can use below command to connect to your original window workstation:
[root@new]# xfreerdp --plugin cliprdr -d [domain] -u [username] -g [w]x[h] 192.x.x.x
2. Install xrdp as RDP Server for Window client. This part would require the Extra Software Repository - EPEL for yum-ing the xrdp package
[root@new]# wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@new]# rpm -ivh epel-release-6-8.noarch.rpm
Then you should refresh your yum repository:
[root@new]# yum repolist
================
epel             Extra Packages for Enterprise Linux 6 - x86_64           11,105
================

Now you can install the xrdp and vnc server:

[root@new]# yum install xrdp tigervnc-server

[root@new]# service xrdp start

Final, you should make those service auto-started after reboot:

[root@new]# chkconfig xrdp on

3. After EPEL, you should be able to install ntfs-3g for NTFS disk access.
[root@new]# yum install ntfs-3g
[root@new]# vim /etc/fstab
/dev/sda2               /mnt/win_d      ntfs-3g rw,umask=0000,defaults 0 0

4. Install Samba to enhance the file transfer for your document

5. Change the mouse scroll like Mac's nature. This would be more convenient for your daily work.
[root@new]#xmodmap -e "pointer = 1 2 3 5 4 7 6 8 9 10"

The XRDP Bug after restart the service

I have installed the XRDP on CentOS for Hadoop Java development. And the Eclipse on CentOS require the RDP to Gnume Desktop for GUI. However, I need all the connection to the same session to work on latest progress. So I added up the static port as session config like below:

[xrdp2]
name=sesman-Xvnc-5910
lib=libvnc.so
username=ask
password=ask
ip=127.0.0.1
port=5910

However, after I restarted the machine. The sesman session shows error on connection. After several try I found the XRDP require an initial session when you try to identify the port of session. Which means when there is no 5910 session on XRDP Service. The above connection will get error. We have to left the config like:

[xrdp1]
name=sesman-Xvnc-New
lib=libvnc.so
username=ask
password=ask
ip=127.0.0.1
port=-1

This config will start up the first session at port 5910 (Default). Then you can successfully identify the session port 5910 as [xrdp2] assigned and get into the session you left after service restart.

There is a way to get your desktop session on localhost:
[xrdp0]
name=sesman-Xvnc-Local
lib=libvnc.so
username=ask
password=ask
ip=127.0.0.1
port=5900

Before restart xrdp, you should change the setting [System]->[Preference]->[Remote Desktop] to open the value of [Allow other User to control your desktop]. The xrdp0 will show the local desktop you have directly work on.

The Fundamental Design for Service Application

Here so called Service Application indicate to background daemon for application such like Request Handle Server, Flow Processor. In Cloud Era, every Service Application should consider these three feature into fundamental design:
1. High Availability: HA has three design level based on the complication and cost. Lowest level is none. Basic Level is Active-Standby. The highest level is Capacity Impact. To reach the Highest Level, it means your system has some kind of scale out ability. However, we have met some terrible design which doesn't control the database transaction and lock logic well. That will introduce incapable to scale out your application and limited your option to Active-Standby Design. Otherwise, all well trained developers should have the ability to implement the application with scale out.

2. Application Resilient: any application would happen the process crash or machine incidentally rebooted. There should be a design to make the service resiliently restarted or elegantly terminated and started by other application. This is not only about the exception handle issue but also the service continuity. However, most modern designed OS has the basic tool to keep this feature on your application.

3. Log Notification: This is must have but each team has different implementation.

QT Creator 5 and Boost Framework on Windows

C++ is a language which brings you benefit from cross-platform and better performance. When we need compile C++ on different plat-form, usually we have to choose different IDE for windows and Linux. However, since QT has support the tool chain from Microsoft Visual C++ (msvc) we found QT might be the neat solution when we need an IDE that support our project on different platform.
For windows, we use boost as the framework for our C++ development (msvc 11 has support the implementation for C++ 11 statndard which is quiet handy that we don't have to build gcc 4.8 on MinGW platform. But for the developer already has code ran on MinGW framework, I believe it is the same handy when you integrated QT with MinGW Tool Chain.) So first thing is down load the boost_1_55_0 into C:\.
It is really easy to get boost installed. Go to C:\boost_1_55_0 and run bootstrap.bat batch file you will get a b2.exe file as your build tool. Then run the b2.exe (you can use b2 --help to find the command arguments for you to specify the customized installation config) The latest built dll would be under "C:\boost_1_55_0\stage\lib" and the header file is under C:\boost_1_55_0\boost. The whole build process should under the windows shell of visual studio 2012 or the compiler which supports C++ 11 standard. You could get the windows shell by using "VS2012 x64 Native Tools Command Prompt".
Hence, please add the follow instruction in your QT .pro file for the project you would like to include boost:

 
INCLUDEPATH+=C:\boost_1_55_0
LIBS+=-LC:\boost_1_55_0\stage\lib\

Then you could use #include<boost\asio.hpp> to see if the compiler work.
This is for framework include. If you want to include the external library such as logger or database, you could right click the Project Icon on left Tree View and add an external library (require a specified dll)

For the debugger on QT, you have to install the cdb.exe from WDK:
http://msdn.microsoft.com/en-us/windows/hardware/hh852365.aspx

If you use the tool chain of visual studio 2012, you should install WDK 8.0.

mRemoteNG with External Tool Set up

Many IT department only support the maintenance for Windows Workstation. So, if your work is mainly on Linux system, here is a great free multi-shell client tool on windows called "mRemoteNG". Although it support plenty of client shell with tabs, we still need the scp tool to upload file from windows workstation to Linux server. "mRemoteNG" provide you a way to invoke the other tools like a plug-in which is a really neat idea. First, you can put your winscp under the root folder of "mRemoteNG".
At the External Tool Properties you can specify the SCP tool's location and the argument you wan to pass into external tools from mRemoteNG like below:

scp://%Username%:%Password%@%Hostname%/

You can use the same method to integrate mRemoteNG and FileZilla.

The Folder Convention for CentOS

Each Linux distribution has their own system folder convention which will let user to figure out the purpose and program deployment layout quick. For example, MySQL's database file would be deployed to "/var/mysql" after you execute mysql_install_db command without specified the instance customized storage path but use default.
Speaking of CentOS, since it is derivative from RedHat distribution, I think it is quiet easy to grab its folder convention from application server development view. In daily operation, the "/var" folder would be the most important place for data storage. "/var" folder contains all the data file or "variable data of system persistent layer" and I believe this convention should be kept from server to server.
The second folder is "/opt" this folder is containing all the application server which is like a small OS inside Linux. I found some server package would assign /opt as their parent of server root folder and the script tool of application server would consider "/opt" is the shell path for scripting included. However, "/opt"s' purpose has most discrepancy from distribution to distribution. But I prefer use "/opt" as a small set of system package. Under "/opt" folder, each server package has its own "/usr", "/bin" and even "/sbin". It likes once you get into "/opt/server", you will get your own world with a dialect to communicate with another system. But sometimes it is pretty hard to configure the program under "/opt" for well function because there might be plenty binary dependency between "/opt" package and "/usr".
If you have built up a server with some auto run daemon or program, the "/etc" folder might be your most accessed folder per day. Surprisingly, "/etc" not only contains important configuration file from OS to Server Container (such as "/my.cnf"). "/etd/init.d" also has a lot of script for daemon and server start up. So you can tread "/etc" like a launch center to configure and regular the behavior of your whole application server from OS to user application at every beginning.
"/usr" is most crucial for all developer. The "/usr" folder would show the character of each Linux Dist. Even for the same distribution, the "/usr" folder might be much different from server edition to desktop edition. All the important compiler, productivity tool and open source software are scattered in the sub-folders of "/usr". If "/bin" and "sbin" are the shell that communicate to your machine, the "/usr" would give you the ability to enhance all your work to the machine. Most open source software are targeted to be deployed into "/usr" and scattered into "/usr/local", "/usr/bin", "/usr/include", "/usr/lib" and "/usr/etc". The famous Linux Package Manager and Deployment software are managing the "/usr" folder with their deliberated protocol and mechanism. And those deployment tools are really helpful for environment set up such as Java or X windows. However, some fresh new package might require you twist the file under "/usr" folder. That means the new package has not been adopted by those deployment tools and you have to dig into "/usr" folder's set up for running the new program normally. Neither, there is no easy way to unplug all those manually installed software from "/usr" with fresh clean up.

CentOS or Window OS Python Environment Setup

We should download the python package "distribute" first which will install easy_install script first. The easy_install is installation tool for "pip". And after pip installed, we can use pip to set up all the package we need such as Django, VirtualEnv or Happybase.
We use windows as example:
First, install the python package on windows and add "C:\Python27" into Environment Path. Second, use the distribute_setup.py to let python package has ability to extend the feature such as happybase or django. You can download the package from:

https://pypi.python.org/pypi/distribute/0.6.49

Un-archive the zip file you will get a folder "dist" and copy to your python installation root folder. Then execute the command to run setup py: "C:\Python27> .\python.exe .\dist\distribute-0.6.49\setup.py install" (You can apply this process to install Django too)
Then you will see a folder "C:\Python27\Scripts" which contains python package manager program for windows. Please execute "C:\Python27\Scripts> .\easy_install.exe pip" for install pip which is the official program to let us down load the package such as happybase.
Final, you could use "pip list" to check how many package or extension on your python environment.

MySQL Related Set up Step such like Xtrabackup

On CentOS 5, we need libaio for MySQL server installation.

$ sudo yum install libaio

$ sudo rpm -ivh MySQL-server-5.5.31-2.rhel5.x86_64.rpm
Preparing...                ########################################### [100%]
   1:MySQL-server           ########################################### [100%]
$ sudo rpm -ivh MySQL-shared-compact-5.5.31-2.rhel5.x86_64.rpm
Preparing...                ########################################### [100%]
   1:MySQL-shared-compat    ########################################### [100%]
$ sudo rpm -ivh MySQL-client-5.5.31-2.rhel5.x86_64.rpm
Preparing...                ########################################### [100%]
   1:MySQL-server           ########################################### [100%]

We have to let MySQL use "Table Per File" mode. For example, the my.cnf mysql section setting is like:

[mysqld]
......
......
innodb_data_file_path = ibdata1:10M:autoextend
innodb_file_per_table = 1
datadir = /var/lib/mysql/datadir
innodb_data_home_dir = /var/lib/mysql/ibdatadir
innodb_log_group_home_dir = /var/lib/mysql/log
tmpdir = /var/lib/mysql/tmp

Then we should use the shell script to make the folder ready:

$cd /var/lib/mysql
$sudo mkdir datadir
$sudo mkdir ibdatadir
$sudo mkdir log
$sudo mkdir tmp
$sudo chown -R mysql:mysql /var/lib/mysql

The fresh New MySQL instance could be create by the following command.

$ sudo mysql_install_db

Alternatively you can run:

$/usr/bin/mysql_secure_installation

which will also give you the option of removing the test databases and anonymous user created by default. This is strongly recommended for production servers. The XtraBackup has different build based on Linux Kernel ang glic version. percona-xtrabackup-2.1.5-680.rhel6.x86_64.rpm is suitable to CentOS 6 percona-xtrabackup-2.1.5-680.rhel5.x86_64.rpm is suitable to CentOS 5 Don't try to upgrade the GLIBC's version in Linux Kernel otherwise the Kernel would be instable by change the Glibc. your can use the Linux command below to check the OS Release edition and MySQL's version:

rpm -qa | grep MySQL
rpm -qa | grep percona
cat /etc/*release
uname -a

  MySQL 5.1 is only supported by XtraBackup 2.0.8. Therefore, for older MySQL we need use previous version's XtraBackup. For MySQL 5.5+, XtraBackup 2.1.5 is OK. Here is the Installation Process:

$ rpm -ivh percona-xtrabackup-2.1.5-680.rhel6.x86_64.rpm
warning: percona-xtrabackup-2.1.5-680.rhel6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID cd2efd2a: NOKEY
error: Failed dependencies:
        perl(DBD::mysql) is needed by percona-xtrabackup-2.1.5-680.rhel6.x86_64
        perl(Time::HiRes) is needed by percona-xtrabackup-2.1.5-680.rhel6.x86_64

You can see the innobackupex.pm perl script require some package for running. Hence we should install the perl packages first:

sudo yum install perl-DBD-MySQL
sudo yum install perl-Time-HiRes

Then the installation of XtraBackup would be successful.

$ sudo rpm -ivh percona-xtrabackup-2.1.5-680.rhel6.x86_64.rpm
warning: percona-xtrabackup-2.1.5-680.rhel6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID cd2efd2a: NOKEY
Preparing...                ########################################### [100%]
   1:percona-xtrabackup     ########################################### [100%]

Just in case that list down the error message for CentOS5.

#CentOS 5 is too old and require more dependency for XtraBackup
$ rpm -ivh percona-xtrabackup-2.1.5-680.rhel6.x86_64.rpm
warning: percona-xtrabackup-2.1.5-680.rhel6.x86_64.rpm: Header V4 DSA signature: NOKEY, key ID cd2efd2a
error: Failed dependencies:
        libc.so.6(GLIBC_2.7)(64bit) is needed by percona-xtrabackup-2.1.5-680.rhel6.x86_64
        libc.so.6(GLIBC_2.8)(64bit) is needed by percona-xtrabackup-2.1.5-680.rhel6.x86_64
        perl(DBD::mysql) is needed by percona-xtrabackup-2.1.5-680.rhel6.x86_64
        rpmlib(FileDigests) <= 4.6.0-1 is needed by percona-xtrabackup-2.1.5-680.rhel6.x86_64
        rpmlib(PayloadIsXz) <= 5.2-1 is needed by percona-xtrabackup-2.1.5-680.rhel6.x86_64

Linux Hardware Information Collection Command

Recently, my most job context is related to Linux CentOS and MySQL with perl and python programming. However, when I install the MySQL server, there are some command could help me get more base information from the remote server that I am dealing with:
Show kernel version and system architecture

uname -a

Show name and version of distribution

head -n1 /etc/issue

Show all partitions registered on the system

cat /proc/partitions

Show RAM total seen by the system

grep MemTotal /proc/meminfo

Show CPU(s) info

grep "model name" /proc/cpuinfo

Show info about disk sda

hdparm -i /dev/sda

And after the hardware information, we use RPM command to review the package list on this machine:

rpm -qa | grep MySQL*
rpm -qa | grep percona*

If the some RPM package is missing from your wonder list, you could use yum command (with internet connection) or download the RPM package (be careful the Kernel version and OS distribution version) and use RPM command to install those dependencies:

rpm -ivh [wonder package]

For example, we need the crontab package for doing some schedule task on specific server. However, the crontabs require the cronie as dependency and cronie require sendmail as dependency and the dependent chaining is going further to procmail. Hence, below is the demonstration for installation of procmail, sendmain and final step with install crontabs related package at once (must have, because they seem to related in a cyclic check, you could not install single one of them without two others)

[root@ServerA ~]# rpm -ivh cronie-1.4.4-7.el6.x86_64.rpm
error: Failed dependencies:
        /usr/sbin/sendmail is needed by cronie-1.4.4-7.el6.x86_64
        dailyjobs is needed by cronie-1.4.4-7.el6.x86_64
[root@ServerA ~]# rpm -ivh sendmail-8.14.4-8.el6.x86_64.rpm
error: Failed dependencies:
        procmail is needed by sendmail-8.14.4-8.el6.x86_64

[root@ServerA ~]# rpm -ivh procmail-3.22-25.1.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:procmail               ########################################### [100%]
[root@ServerA ~]# rpm -ivh sendmail-8.14.4-8.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:sendmail               ########################################### [100%]
[root@ServerA ~]# rpm -ivh cronie-anacron-1.4.4-7.el6.x86_64.rpm crontabs-1.10-33.el6.noarch.rpm cronie-1.4.4-7.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:crontabs               ########################################### [ 33%]
   2:cronie                 ########################################### [ 67%]
   3:cronie-anacron         ########################################### [100%]

The Design Strategy of Distributed System Based on HTTP World

Today's distributed systems are based on internet and HTTP connection. Which means the private protocols other than TCP/IP connection are less supported and non-open standard. And beyond the Level 4 Network concept framework, HTTP and SOAP definitely dominate the way in transmission among those distributed systems. Here I wanna discuss several aspects of the technology design strategy existed in those distributed system. Before we expand the discussion, here are some terminology that lay out the system development.

Three-Tier System Architecture: In traditional distributed system, three-tier system has UI Layer, AP Layer and Data Layer. In Web-like distributed system, the UI Layer involve the browser and web page container. The AP Layer might involve a lot of SOAP-like service. If it's necessary, we sometimes have to dig into TCP/IP connection for communicating with legacy system which has no HTTP protocol OS support. The Data Layer is Database or File System on legacy system.

Development Stack: The technology we use to develop the distributed system. Corresponding to the Three-Tier system, the web page container and SOAP-like service management would rely on server we choose such as IIS, Tomcat, Jetty etc. You can see the application server could determine the development framework and languages - .Net and Java. There are some other popular application frameworks like Rail, PHP, Djongo would be based on Apache server. Those application frameworks have different Web UI rendering strategy, SOAP or RESTful management model and Data Persistent Framework for adoption in your project.

The web socket server implementation on Python

Recently, the Java Applets Sand Box were breached by hacker and cause a lot of security event on web. Therefore, the last sand box would be the browser itself. (If there is no trust-able sand box, I don't know how to build up a reliable infra on WWW.) And I used to write the TCP/IP Socket in Applet for real-time application on web. Therefore I turned into a new standard which is still in draft of W3C - Web Socket.

I have learned there is a client implementation in JavaScript for web socket - socket io. And also there are several server implementation on web socket including the Python. However, I found this implementation is quiet bit experimental-wise. First, its base library is called gevent-socketio which is the web socket server implementation that depends on gevent, greenlet and libevent. And those three libraries are pretty nasty when I try to build the package environment including them. First I have to let Python learn the compiler's setting to build libevent and hence the gevent and greenlet would be installed successfully. But I quit this way because my window's machine is too clumsy and no more 10 GB for visual studio as a compiler. So I choose the web site http://www.lfd.uci.edu/~gohlke/pythonlibs/ which provide the built binary for windows installation.

After I install the gevent and greenlet, I can easily use pip to install the gevent-websocket. (off course you should  install the easy_install tool package first which is still a binary installation from https://pypi.python.org/pypi/setuptools. After all, we are dealing with windows without apt-get). The package of gevent-websocket is the protocol implementation for web socket standard and pretty easy to be install once you have easy_install set up.

So my Python Interpreter can run the script without error now. It's time to try the web socket server of Python. Since I couldn't afford a Mac or event Chrome book, I think this is the only way that dive into Python's world.

Javascript Object-Oriented coding style and Binding technique for non-blocking context

Javascript is a powerful language. Since I learn the node.js, I found the non-blocking style's Javascript coding could bring us some advantages on interactive and agile development process. However, behind those advantages there are some tricks to whom we should give attention while we write the code.

Objected-Oriented Javascript

Assuming we have a data model called ClassA, and the code is below:

var ClassA = function (client_name){
    this.name = client_name;
    var property1 = "this is the property 1 value of " + this.name;
    this.selfIntroduction = function (){
     return "Hello World! My name is " + this.name;
 };
    this._getProperty1 = function(){
     return property1;
 };
    this.getProperty1 = function(){
     return this.property1;
 };

    this.property2 = "this is the property 2 value of " + client_name;
    this._getProperty2 = function(){
     return property2;
 };
    this.getProperty2 = function(){
     return this.property2;
 };
};

var property1 = "Are you sure this is the property1 you want?";//global variable
var property2 = "Are you sure this is the property2 you want?";//global variable
// allocate the object of ClassA
var obj = new ClassA("user1");

console.log("client name: " + obj.name);
console.log("client var p1: " + obj.property1);
console.log("client private _get p1: " + obj._getProperty1());
console.log("client public   get p1: " + obj.getProperty1());
console.log("client this p2: " + obj.property2);
console.log("client private _get p2: " + obj._getProperty2());
console.log("client public   get p2: " + obj.getProperty2());

we output the obj and get this result:

client name: user1
client var p1: undefined
client private _get p1: this is the property 1 value of user1
client public   get p1: undefined
client this p2: this is the property 2 value of user1
client private _get p2: Are you sure this is the property2 you want?
client public   get p2: this is the property 2 value of user1

Here has some points require our attention
1. In ClassA, the "var" variable is private variable for "obj". We can only use private _get method without "this." as prefix in function to address property 1.
"obj.getProperty1()" is in totally wrong way to address property1. "this.property1" and "var property1" are absolutely different things.

2. Proper2 has declairation of prefix "this." which expose property2 as a public variable.
We could use "obj.property2" or public get method with "this." as prefix in function for addressing property2.

From this two observation, we could clearly be aware of the strictly discrimination from private variable to public variable in Javascript.

3. The most tricky part and also showing the odd behavior of Object-Oriented Javascript is the "obj._getProperty2()" with unexpecting result from global variable.
We get a response that would never happen while we are working on the C++, JAVA or C#.

And it will bring us two terms in Javascript - anonymous function and Bubble-up scoping.
First, "this._getProperty2" is a public reference just like "this.property2". However, this public reference points to an anonymous function
and when we apply "obj._getProperty2()" to "console.log", it is actually an inline function like:

console.log("client private _get p2: " + (function(){return property2;})());
//"(function(){return property2;})()" means directly execution of this anonymous function after we have implemented it.
//this technique is quite often used as constructor in Javascript object.

Apparently, no one has claim the property2 in this line, neither the "console.log". Only one guy has claimed and given the memory space to property2 - "global wise" i.e. node.
Javascipt has the behavior like HTML. They would bubble up for searching the variable reference.
Therefore, we could never get "undefined" but in totally wrong value without acknowledge of it, which is a nightmare for debugging.

For preventing us from this situation, we should alway use "this." to refer the public variable inside the class's public method.
Even though for class construction or encapsulation, there might be some private variable in our class, well-documented class layout and "undefined" detection can avoid the
confusion from bubble-up scoping. For example, "obj.property1" and "obj.getProperty1()" these public reference accessing all show "undefined" with consistent results
and they won't resolve the "property1" in global scope. What I mean is to rather declaire the "var" in the constructor-"function ClassA"-as possible and only
expose variable to public "this." when you really know what you are doing. On the otherhand, accessing the variable through "this." in Class's public method as possible.
Once you want to access the private variable in public refered function, you better know what you are doing.

Non-blocking Callback and Variable Resolution Issue

So far, we have stressed out some neccessary knowledge in Object-Oriented Javascript. Next, we bring the content further to multi-context management.
Suppose we have a function "main" which exploit some non-blocking api for interactive and dynamic linking in Javascript.

function main(){
    var obj2 = new ClassA("user2");
    ClassA.prototype.callbackHandler = nonblock_callback;//dynamically link a function to obj2
    var socket = new fake_io();
    socket.send(obj2.callbackHandler);
    console.log("Has issued a request to server");
}
function nonblock_callback(event){
    console.log("callback has been triggered by " + event);
    console.log("this is the callback event handler of " + this.name);
}
function fake_io(){
    this.send = function(callback){
        setTimeout(function(){callback("my event from fake_io");}, 2000);
    };
}
main();//execute

The result is below and be careful of last line in output:

Has issued a request to server
callback has been triggered by my event from fake_io
this is the callback event handler of undefined

Here we have applied some technique "prototype" in Javascropt. This is a basic infrastructure for Javascript's inheritance and overriding.
(sorry! I don't know how to do overloading. If you did, I'll appreciate your sharing).
Although we construct obj2 first, "ClassA.prototype" still give us a way to change the class layout for obj2 with dynamically attaching a handler function "nonblock_callback".
(We can only use "obj2.callbackHandler = nonblock_callback" for adding a function to obj2, but my purpose is to emphasize the ability of "prototype".
We must use prototype with discretion especially in this scenario which obj1 is also affected by "ClassA.prototype".)

After the delightful usage of "prototype", let us review the last line of response. What happened to this handler of user2?
Ok, let me rephrase the question: Who own your context while callback is triggered? (actually these are two different question, but last one is real key.)
I will put this question away and do a small experiment first. Put an extra line inside the "main()" function for declaring a public property "this.name".
Execute main() again.

function main(){
    this.name = "Ha Ha! I am the Javascript devil.";
    ......
 ......
}

Well, you have seen what I mean. Since we would apply Object-Oriented Javascript and non-blocking style.
This issue should be understood thoroughly by all programmers before implementation.
Firt, we consider the inline function while the time after two seconds was invoked by setTimeout.

main(){
    this.name = "Ha Ha! I am the Javascript devil.";//without this line "this.name" would be undefined
    (function(){nonblock_callback("my......"){console.log......; ...... + this.name);}})();
}

Yes, the whole context showed either this.name is undefined or we have the value whose owner is totally different from our expecting.
Javascipt has two apis named "call" and the other is "apply". They provide the same function which let us insert the object reference on callback function.
For example, in fake_io, we change the callback("my event from fake_io") into callback.apply(obj, ["the event_handler invoked by "]).
Therefore, the source code of fake_io would become:

function fake_io(){
    this.send = function(callback){
        setTimeout(function(){
     callback.apply(obj,["my event from fake_io"]);
 }, 2000);
    };
}

Now, we get resolution of "user1" in callback function. Hence, we could understand the power of "call" and "apply".
(please google them for asking the difference from these two apis).
But, once we change the callbask into callback.apply(obj2,["my event from fake_io"]), you might anticipate what problem exists in this code.

/home/brianko/JavascriptOO.js:54
     callback.apply(obj2,["my event from fake_io"]);
     ^
ReferenceError: obj2 is not defined
    at Object._onTimeout (/home/brianko/JavascriptOO.js:54:6)
    at Timer.ontimeout (timers.js:85:39)

Yes, the garbage collection has recycled our obj2 in main() and the context invoked by setTimeout has no vision about where is our obj2.
We need smarter way to do that and this technique is the knack called Javascript binding.
First, at the very beginning of code, we create a binding wrapper which is actually a Javascript function closure.

var callback_binding = function(obj, handler){
    var _self = obj;
    var _funcptr = handler;
    return function(){
     return _funcptr.apply(_self, arguments);
    };
}

Then, we change the deliberation of fake_io in main().

function main(){
    .........
    ClassA.prototype.callbackHandler = nonblock_callback;//dynamically link a function to obj2
    var callback = new callback_binding(obj2, obj2.callbackHandler);//binding
    var socket = new fake_io();
    socket.send(callback);
    .........
}
function fake_io(){
    this.send = function(callback){
        setTimeout(function(){callback("my event from fake_io");}, 2000);
    };
}

And the fake_io is the same as original one. Genuinely speaking, we don't have to dig into the fake_io which might be the component you bought from outside.
From the documentation, we know the fake_io.send() is non-blocking style and we use binding technique to wrap our callback function in main(), where we deliberate the fake_io.
Then execute main() function, your callback function will correctly resolve "this.name" as "user2".

While we are developing a mass project with some object-oriented Javascript technique, this is the knack to unleash the non-blocking power in Javascript.
Have fun!
Below is our code in JavascriptOO.js for running on node.js.

var callback_binding = function(obj, handler){
    var _self = obj;
    var _funcptr = handler;
    return function(){
 return _funcptr.apply(_self, arguments);
    };
}

var property1 = "Are you sure this is the property1 you want?";
var property2 = "Are you sure this is the property2 you want?";

var ClassA = function (client_name){
    this.name = client_name;
    var property1 = "this is the property 1 value of " + this.name;
    this.selfIntroduction = function (){
     return "Hello World! My name is " + this.name;
 };
    this._getProperty1 = function(){
     return property1;
 };
    this.getProperty1 = function(){
     return this.property1;
 };

    this.property2 = "this is the property 2 value of " + client_name;
    this._getProperty2 = function(){
     return property2;
 };
    this.getProperty2 = function(){
     return this.property2;
 };
};

var obj = new ClassA("user1");
/*
console.log("client name: " + obj.name);
console.log("client var p1: " + obj.property1);
console.log("client private _get p1: " + obj._getProperty1());
console.log("client public   get p1: " + obj.getProperty1());
console.log("client this p2: " + obj.property2);
console.log("client private _get p2: " + obj._getProperty2());
console.log("client public   get p2: " + obj.getProperty2());

console.log("anonymous p2: " + (function(){return property2;})());
*/

function main(){
    this.name = "Ha Ha! I am the Javascript devil.";
    var obj2 = new ClassA("user2");
    ClassA.prototype.callbackHandler = nonblock_callback;//dynamically link a function to obj2
    var callback = new callback_binding(obj2, obj2.callbackHandler);
    var socket = new fake_io();
    socket.send(callback);
    //socket.send(obj2.callbackHandler);
    console.log("Has issued a request to server");
}
function nonblock_callback(event){
    console.log("callback has been triggered by " + event);
    console.log("this is the callback event handler of " + this.name);
}
function fake_io(){
    this.send = function(callback){
        setTimeout(function(){
     callback("my event from fake_io");
 }, 2000);
    };
}
/*function fake_io(){
    this.name = "You better think about it";
    this.event = "my event from fake_io";
    this.send = function(callback){
        setTimeout(function(){callback(this.event);}, 2000);
 //callback(this.event);
    };
}*/
main();//execute

Composite Pattern on Web Service

In recent day, I have seen for some requirements that the system has been adopted to multiple devices other than web such as mobile, tablet etc. However, I found some system has their business logic embedded into some web page rendering code. As a Enterprise Distributed System Developer, there always turn out a question which software programmer might not have to encounter - Should I implement an Interface on Web service?

People like you might think : what's difference? Since you have decouple the modularity  through an Interface, so you could just make reference to the module in web page or some other project that consume this module as a client. But here are the things, first this Interface contains an composite pattern which provide so many operations we want them to be hidden from web UI which means the UI team doesn't have to understand the operation detail. Second is the crucial reason that the operations require spec change all the time and we don't want to publish or deliver this module with the consequence of impacting the UI (they might have to recompile for adopting new feature of the business logic that just has been re-design and re-implement).

Hence, the Web Service-lized Composite Pattern gives us the benefit that we can expose whole bunch powerful services within Enterprise Distributed System and the clients from the other department or business domain don't have to learn about the detail. Even more, as long as the interface hasn't been re-org or re-structured, the business logic inside could adopt the new governor regulation, company policy or business management methodology without impact the UI. However, things are not as simple as I thought that just move the module of composite class onto web service. There are some point of view we have to consider about:

1. Are all the operation Thread-Safe? Assumable speaking your composite class can execute the complex business context. Then it might handle the multiple transaction between database. Or the class utilize the vendor's library which manage some file system or TCP/IP connection to the external legacy system. There always comes surprises within the library of black box especially when the module deal with so many business contexts as the system's feature advanced. Usually I will use the ajax to simulate the simultaneous request to a composite class that has just been implement behind web service. We need a developer with substantial business knowledge who is hard to be recruited for tracing the any resource contention or transaction lock inside the class. You could wrap the composite class with singleton pattern for managing the potential trouble of intrinsic multi-threaded web request. However, this superficial solution would bring the next question.

2. Would the labor operation drag down the performance and stability of Web Application Server severely? Web Server can not do every thing for us while the operation is really laboring and time consuming. While you have a resident daemon in OS that work for a laboring job, the web interface on this operation for monitoring, event dispatching and notification might not a good design due to the object management and scope would be handed out to web server not your daemon. Web server has its limit and we shouldn't treat it like a dummy daemon with all benefit about web access.

3. Do you have good solution to deal with Stateless Connection on State-full Transaction Management? Business context might be long and synchronization required. If you use your composite class within an application which provide the state management of data model. Then you should prepare that any context cutting and state synchronization effort would emerge in coming future.

4. How you handle the Authentication and Authority Logic rely on Domain or other Client Certificate Technique? These crucial features are resident in whole business context. Separating the composite class bring you to verify the authentication and authority between client library and web service again. For some enterprise systems, the principal-deputy relationship, surrogate operation and audit regulation might be your major burden while you want to pull out some business context for flexibility and accessibility.

Those issues should be considered at the beginning of the system and architecture design. Unfortunately, many projects are implemented before they realize where are the composite pattern. Even worse, the UI's code and business Logic have been entangled that require a lot of re-factor work before we extricate the composite class to web service.

Cloud Foundry

IaaS has been proved it is a workable service model. I have seen the Openstack built up my Ubuntu virtual machine in a few minutes. Therefore I can easily create more resource for system development and test.

SaaS is an emergent model for software vendor who can charge their customer based on the usage of software rather than the installation licence. However, even with the support of IaaS, SaaS still face the huge maintain cost on system configuration and backup strategy. Although IaaS brought us a better way on virtualization management, we still have to install, configure and trouble shooting on the system's essential utility or framework such as JVM, HTTP server, Database etc. Hence, the PaaS can fill this gap and bring the SaaS vendor more efficient development cycle especially for those SaaS vendor whose client base requires more customized function.

Cloud Foundry is an opensource PaaS provided by the most famous virtualization solution provider - VMWare. Although Google has the Google App Engine for SaaS vendor. VMWare's solution would be really attractive to the company self-owning private cloud. Most Company who has applied virtualization technology can enhance the infrastructure into IaaS smoothly under the assistance of adequate tool. However, it is really difficult for most company to define or create a PaaS on their private cloud. Hence, most developers are still struggling on designing the platform building block, essential utility configuration, backup, deployment strategy and maintenance. The shortage of PaaS would be the consolidation of platform which might not fit into the certain software really well. Under this circumstance, we might have to customized some configuration or assemble the essential utility for special requirement. Nonetheless, the customized utilities could still be part of our PaaS co-working with the other standardized framework or utilities.

Dependency Management

In the system development, that is really usual that multiple members would join their project to work out a specific function for the coming demand on this system. As the past paradigm which focused on the software and package aspect about the system, the all dependency would be considered about at the early design stage and we have already whole bunch tools that help us to control on those "static binding dependency" between the different binary library files. The compiler is the first stop to be the static binding dependency checker for our software and package. And the programming language infrastructure like C++ has introduced the namespace concept into our practice and give the developers and IDE a good foundation to manage the static binding dependency. A good static dependency design brings benefit on the system flexibility in the future. Therefore, there are so many useful design pattern about how to depict the hierarchy of classes. A good design pattern would have percipient about future change and coordinate libraries in a modest way.
However, just like the beginning of this post, the static binding only solve the internal coupling issue of a set of binary libraries. There are more issues when we combine all those projects into a system. We can categorizing those issues into two genre. One is the issue at early binding. It's pretty common that our program is running on a runtime like .NET or JVM and container like IIS, Tomcat, WPF or even a windows service. The runtime and container provide the variety and useful core libraries for our application. Hence, we write the configuration file for deploying our binary file quite often. Those configuration file is the prime communication way to those runtime and container and tell them what kind of basic service we want those foundation to provide such as authentication method, session control, type including or Resource location. Once the configuration has trouble with the binary files we've deployed or the runtime and container's ability. The early binding activity would alert an error at the initiation of the executable file when Operation system load the whole application into memory. Those executable files such as w3wp, java.exe are the most common container and runtime which require a strong background knowledge about their configuration or command line parameters. Early binding issues require the modern IDE or some trouble shooting skill to dig them out when we try to deploy the application on new machine.
The second genre of dependency management issue other than static binding is the lately binding which is the hardest part without a reliable tool when we deploy the project that we don't familiar with. So called lately binding means that we might never be aware of some type or class implementation loss at deployment until the actual request or running application went through those particular section of our source code. The lost library or class are only initiated when the application ask for. Those libraries or classes are dynamically loaded or reflected by the program when it's necessary. Which means there might be no adequate audit or document that recalls that weak points of the deployment activity. The dynamic loading mechanism is pretty advanced feature in modern system development. Although the blind spot at deployment activity would jeopardize our online application, the flexibility and replaceable binary at runtime still be really attractive to a lot of developers adopt this mechanism into their system design.
The three dependency management aspects mentioned above are all talking about the dependency inside the same process image of a specific memory section that operation system has allocated. But there is a dependency issue that is quiet common at distributed system project. The issues are addressed about the remote service dependency like web service or client-server dependency. At usually circumstance, the consumer (i.e. client) require a specific service provided by a non-existing or accidentally missing server. The occasion was generated by the remiss on deployment activity. Like the missing of dynamical loaded libraries, the possibility of sending the request to a vacant service entry would rise the stake of a certain exception that damages our online application. However, this is due to the neglect of some specific service for the system and the remedy is pretty easy if we have dealt with those exception neatly, just start the remote service. The trade-off between flexibility and integrity of deploying should not be skipped while system analysis. And the balance of flexibility and integrity can also be affected by or do impact to the design of static binding mutually. But this big question is belonging to the domain knowledge among how we isolate the most often changed part in some certain applications. This philosophy perspective has beyond the scope of this article.

Is this the ultimate destination?

I was a wordpress user for a while. However, my purpose with wordpress is to practice build up a web site with technical article collection and notes. After tried XAMPP and AWS to build wordpress, I found that my multiple small blog sites have accumulated certain amount information and articles. Besides wordpress, I also try the joomla for recording my English study and Ubuntu set up note.
Therefore, now is the time to look for a permanent place to put all my stuff in. And I hope Google would be my last destination. I have tired about migrating MySql Database file and PHP reconfiguration......