Elasticsearch cluster status

Posted on 2017-03-21 | Edited on 2020-09-17 | In search

Terminology

An Elasticsearch cluster is made up of one or more nodes.
Each of these nodes contains indexes which are split into multiple shards.
Elasticsearch makes copies of these shards called replicas.
These (primary) shards and replicas are then placed on various nodes throughout the cluster.

Status

Cluster Health API

curl -XGET http://192.168.3.11:9200/_cluster/health

{
  "cluster_name" : "testcluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 5,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 50.0
}

RED:
Damnit. Some or all of (primary) shards are not ready.
YELLOW:
Elasticsearch has allocated all of the primary shards, but some/all of the replicas have not been allocated.
GREEN:
Great. Your cluster is fully operational. Elasticsearch is able to allocate all shards and replicas to machines within the cluster.

Request Parameters

level
Can be one of cluster, indices or shards. Controls the details level of the health information returned. Defaults to cluster.
wait_for_status
One of green, yellow or red. Will wait (until the timeout provided) until the status of the cluster changes to the one provided or better, i.e. green > yellow > red. By default, will not wait for any status.
wait_for_no_relocating_shards
A boolean value which controls whether to wait (until the timeout provided) for the cluster to have no shard relocations. Defaults to false, which means it will not wait for relocating shards.
wait_for_active_shards
A number controlling to how many active shards to wait for, all to wait for all shards in the cluster to be active, or 0 to not wait. Defaults to 0.
wait_for_nodes
The request waits until the specified number N of nodes is available. It also accepts >=N, <=N, >N and <N. Alternatively, it is possible to use ge(N), le(N), gt(N) and lt(N) notation.
timeout
A time based parameter controlling how long to wait if one of the wait_for_XXX are provided. Defaults to 30s.
local
If true returns the local node information and does not provide the state from master node. Default: false.

1	curl -XGET http://192.168.3.11:9200/_cluster/health?wait_for_status=yellow&timeout=50s

Elasticsearch demo

Posted on 2017-03-20 | Edited on 2020-09-17 | In search

Create a new index

1
2
3

curl -XDELETE http://192.168.3.11:9200/policy

curl -XPUT http://192.168.3.11:9200/policy

A new field mapping

curl -XPOST http://192.168.3.11:9200/policy/fulltext/_mapping -d'
{
    "fulltext": {
            "_all": {
            "analyzer": "ik_max_word",
            "search_analyzer": "ik_max_word",
            "term_vector": "no",
            "store": "false"
        },
        "properties": {
            "content": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_max_word",
                "include_in_all": "true",
                "boost": 8
            }
        }
    }
}'

Add data

curl -XPOST http://192.168.3.11:9200/policy/fulltext/1 -d'
{"content":"2017年1月1日起全国实行国Ⅴ排放标准"}
'
curl -XPOST http://192.168.3.11:9200/policy/fulltext/2 -d'
{"content":"2017年1月1日起实施:车内空气质量强制达标"}
'
curl -XPOST http://192.168.3.11:9200/policy/fulltext/3 -d'
{"content":"2017年试行新能源汽车碳配额管理"}
'
curl -XPOST http://192.168.3.11:9200/policy/fulltext/4 -d'
{"content":"新能源车生产企业准入规则 有望于2017年正式实施"}
'
curl -XPOST http://192.168.3.11:9200/policy/fulltext/5 -d'
{"content":"电池行业新规范 有望于2017年全面执行"}
'
curl -XPOST http://192.168.3.11:9200/policy/fulltext/6 -d'
{"content":"新能源补贴新政即将出炉 2017年将实施"}
'
curl -XPOST http://192.168.3.11:9200/policy/fulltext/7 -d'
{"content":"《外商投资产业指导目录》修订版将执行 合资车企股比不会放开"}
'

Have a search

curl -XPOST http://192.168.3.11:9200/policy/fulltext/_search  -d'
{
    "query" : { "match" : { "content" : "能源" }},
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "content" : {}
        }
    }
}
'

Check engine status

1
2
3

curl -XGET http://192.168.3.11:9200/policy/_analyze?text=2017&tokenizer=ik_max_word

curl -XGET http://192.168.3.11:9200/_cluster/health?pretty=true

Hello elasticsearch

Posted on 2017-03-19 | Edited on 2018-12-16 | In search

Install

curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.2.tar.gz

sudo tar -xzvf elasticsearch-5.2.2.tar.gz

sudo chown -R pi:pi elasticsearch-5.2.2

If not change directory’s owner, user command sudo -E ./bin/elasticsearch execute the script, or the java path can’t be find.

Config

./bin/elasticsearch
dispatch memory

1	ES_JAVA_OPTS="-Xms70m -Xmx70m"

./config/elasticsearch.yml
set bootstrap to single machine mode

cluster.name: elastic-pi

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

network.host: 0.0.0.0
http.port: 9200

#http.cors.enabled: true  
#http.cors.allow-origin: "*"

If exception happen like this
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
In single machine mode looks like no this problem

1 2	sudo echo "vm.max_map_count=262144" >> /etc/sysctl.conf sudo sysctl -p

Plugin

elasticsearch-analysis-ik

Pharse process plugin
release to plugin directory

1
2
3

curl -O https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.2.2/elasticsearch-analysis-ik-5.2.2.zip

unzip elasticsearch-analysis-ik-5.2.2.zip

head

install npm/ nodejs/ elasticsearch-head

Exception

org.elasticsearch.indices.IndexClosedException: closed

delete the origin index and rebuild index, it works.

Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME

make sure java be right configed, and current user has privilege to use elasticsearch/ environment variable

Operate

start
nohup ./bin/elasticsearch&
./bin/elasticsearch -d
stop
ps -ef|grep elastic
kill -9 51271

Hello hbase

Posted on 2017-03-15 | Edited on 2018-12-16 | In db

Download Configure

sudo curl -O http://mirrors.hust.edu.cn/apache/hbase/1.3.0/hbase-1.3.0-bin.tar.gz

sudo tar -xzvf hbase-1.3.0-bin.tar.gz

sudo vim conf/hbase-env.sh
 > export JAVA_HOME=/usr/lib/jdk1.8.0_121
 > export HBASE_CLASSPATH=/usr/lib/hbase-1.3.0/lib

sudo bash bin/start-hbase.sh

more logs/hbase-root-master-raspberrypi.log

Have a try

bin/hbase shell

create 'PageViews', 'info'

list

descibe 'PageViews'

put 'PageViews', 'rowkey1', 'info:page', '/mypage'

put 'PageViews', 'rowkey1', 'info:count', '7'

get 'PageViews', 'rowkey1'

put 'PageViews', 'rowkey2', 'info:page', '/myotherpage'

put 'PageViews', 'srowkey2', 'info:page', '/myotherpage'

scan 'PageViews', {STARTROW=>'r', ENDROW=>'s'}

scan 'PageViews', {STARTROW=>'r'}

Inverse function

Posted on 2017-03-14 | Edited on 2020-06-16 | In python

ϕ^{- 1} (y) = \frac{1}{9} y^{3}

$\phi^{-1}(y) = \frac{1}{9}y^{3}$

↓

$\downarrow$

ϕ (x) = 9 x^{\frac{1}{3}}

$\phi(x) = 9x^{\frac{1}{3}}$

#!/usr/bin/python
import numpy as np
import matplotlib.pyplot as plot
N = 10000
X0 = np.random.rand(N)
X1 = 3*X0
Y = np.power(9*X1, 1.0/3)
t = np.arange(0.0, 3.0, 0.01)
y = t*t/9
plot.plot(t, y, 'r-', linewidth=1)
plot.hist(Y, bins=50, normed=1, facecolor='green', alpha=0.75)
plot.show()

Random event

Posted on 2017-03-11 | Edited on 2020-06-16 | In math

All sample’s collection be called sample zone.

Ω_{1} = {1, 2, 3, 4, 5, 6}

$\Omega_1=\{ 1, 2, 3, 4, 5, 6 \}$

Ω_{2} = {t | t \geq 0}

$\Omega_2=\{ t \quad| t\ge0\}$

Contain relation

\emptyset \subset A \subset B

$\emptyset \subset A \subset B$

A \subset B B \subset C \Rightarrow A \subset C

$A \subset B \quad B \subset C \Rightarrow A \subset C$

Equal realtion

A \subset B & & B \subset A

$A \subset B \quad \&\& \quad B \subset A$

Events sum

A_{1} ⋃ A_{2} . . . ⋃ A_{n}

$A_1 \bigcup A_2 ... \bigcup A_n$

⋃_{i = 1}^{n} A_{i}

$\bigcup_{i=1}^nA_i$

Events product

A_{1} ⋂ A_{2} . . . ⋂ A_{n}

$A_1 \bigcap A_2 ... \bigcap A_n$

⋂_{i = 1}^{n} A_{i}

$\bigcap_{i=1}^nA_i$

Mutex events

A B = \emptyset

$AB = \emptyset$

Oppose events

A B = \emptyset & & A ⋃ B = Ω

$AB = \emptyset \quad \&\& \quad A \bigcup B = \Omega$

A = \bar{B} B = \bar{A}

$A = \overline B \quad B = \overline A$

Events difference

A - B = A \bar{B} = A - (A B)

$A - B = A \overline B = A - (AB)$

Complete event group

⋃_{i = 1}^{n} A_{i} = Ω & & A_{i} ⋂ A_{j} = \emptyset (i \neq j)

$\bigcup_{i=1}^nA_i=\Omega \quad \&\& \quad A_i \bigcap A_j=\emptyset (i \neq j)$

Quicksort compare analysis

Posted on 2017-03-10 | Edited on 2020-06-16 | In algorithm

Quicksort user $~2NlnN$ compares to sort
an array of N distinct keys on average
(one-sixth that many exchanges)

Proof

Let $C_N$ be the average number of compares need to sort N items
with distinct values.
We have $C_0=C_1=0$ for $N>1$

Express the average compare

C_{N} = N + 1 + (C_{0} + C_{1} + . . . + C_{N - 2} + C_{N - 1}) / N + (C_{N - 1} + C_{N - 2} + . . . + C_{0}) / N

$C_N = N + 1 + (C_0+C_1+...+C_{N-2}+C_{N-1})/N + (C_{N-1}+C_{N-2}+...+C_0)/N$

cost of partitioning
N-1 compares two partition with flag
1 more compare in each part breakpoint compare

left subarray sort
$(C_0+C_1+...+C_{N-2}+C_{N-1})/N$
right subarray sort
$(C_{N-1}+C_{N-2}+...+C_1+C_0)/N$

Rearrange terms

Multiplying by N

N C_{N} = N (N + 1) + 2 (C_{0} + C_{1} + . . . + C_{N - 2} + C_{N - 1})

$NC_N = N(N+1) + 2(C_0+C_1+...+C_{N-2}+C_{N-1})$

Subtracting with same equation of N-1

N C_{N} - (N - 1) C_{N - 1} = N (N + 1) - (N - 1) N +

$NC_N - (N-1)C_{N-1} = N(N+1)-(N-1)N +$

2 (C_{0} + C_{1} + . . . + C_{N - 2} + C_{N - 1}) - 2 (C_{0} + C_{1} + . . . + C_{N - 2})

$2(C_0+C_1+...+C_{N-2}+C_{N-1}) - 2(C_0+C_1+...+C_{N-2})$

\Rightarrow

$\Rightarrow$

N C_{N} - (N - 1) C_{N - 1} = 2 N + 2 C_{N - 1}

$NC_N - (N-1)C_{N-1} = 2N + 2C_{N-1}$

Each item dividing by N(N+1) leaves

C_{N} / (N + 1) = C_{N - 1} / N + 2 / (N + 1)

$C_N/(N+1) = C_{N-1}/N + 2/(N+1)$

Add same equation from 1

C_{N} / (N + 1) = C_{N - 1} / N + 2 / (N + 1)

$C_N/(N+1) = C_{N-1}/N + 2/(N+1)$

+

$+$

C_{N - 1} / (N) = C_{N - 2} / (N - 1) + 2 / (N)

$C_{N-1}/(N) = C_{N-2}/(N-1) + 2/(N)$

\dots

$\cdots$

C_{3} / (4) = C_{2} / 3 + 2 / 4

$C_3/(4) = C_2/3 + 2/4$

+

$+$

C_{2} / (3) = C_{1} / 2 + 2 / 3

$C_2/(3) = C_1/2 + 2/3$

\Rightarrow

$\Rightarrow$

\frac{C_{2}}{3} + \frac{C_{3}}{4} + . . . + \frac{C_{N - 1}}{N} + \frac{C_{N}}{N + 1} =

$\frac{C_2}{3} + \frac{C_3}{4}+...+\frac{C_{N-1}}{N} + \frac{C_N}{N+1}=$

\frac{C_{1}}{2} + \frac{C_{2}}{3} + . . . + \frac{C_{N - 2}}{N - 1} + \frac{C_{N - 1}}{N} + \frac{2}{3} + \frac{3}{4} + . . . + \frac{2}{N} + \frac{2}{N + 1}

$\frac{C_1}{2} + \frac{C_2}{3}+...+\frac{C_{N-2}}{N-1} + \frac{C_{N-1}}{N} + \frac{2}{3} + \frac{3}{4}+...+\frac{2}{N} + \frac{2}{N+1}$

\Rightarrow

$\Rightarrow$

\frac{C_{N}}{(N + 1)} = \frac{C_{1}}{2} + \frac{2}{3} + \frac{3}{4} + . . . + \frac{2}{N + 1}

$\frac{C_N}{(N+1)} = \frac{C_1}{2}+\frac{2}{3} + \frac{3}{4}+...+\frac{2}{N+1}$

\Rightarrow

$\Rightarrow$

C_{N} \sim 2 (N + 1) (\frac{1}{3} + \frac{1}{4} + . . . + \frac{1}{N + 1})

$C_N \sim 2(N+1)(\frac{1}{3}+\frac{1}{4}+...+\frac{1}{N+1})$

Further integration

\frac{1}{3} + \frac{1}{4} + . . . + \frac{1}{N + 1} \sim l n (N)

$\frac{1}{3}+\frac{1}{4}+...+\frac{1}{N+1} \sim ln(N)$

\Rightarrow

$\Rightarrow$

C_{N} \sim 2 N l n N \approx 1.39 l g N

$C_N \sim 2NlnN \approx 1.39lgN$

PS.

1 + \frac{1}{2} + \frac{1}{3} + . . . + \frac{1}{N} > l n (1 + 1) + l n (1 + \frac{1}{2}) +

$1+\frac{1}{2}+\frac{1}{3}+...+\frac{1}{N}>ln(1+1)+ln(1+\frac{1}{2})+$

. . . + l n (1 + \frac{1}{N - 1}) + l n (1 + \frac{1}{N})

$...+ln(1+\frac{1}{N-1})+ln(1+\frac{1}{N})$

\Rightarrow

$\Rightarrow$

1 + \frac{1}{2} + \frac{1}{3} + . . . + \frac{1}{N} > l n [2 \times \frac{3}{2} \times \frac{4}{3} \times . . . \frac{N}{N - 1} \times \frac{N + 1}{N}] \approx l n (N)

$1+\frac{1}{2}+\frac{1}{3}+...+\frac{1}{N} > ln[2 \times \frac{3}{2} \times \frac{4}{3} \times ...\frac{N}{N-1} \times \frac{N+1}{N}] \approx ln(N)$

Euclidean algorithm

Posted on 2017-03-09 | Edited on 2020-09-17 | In math

int gcd(int a,int b) {
    if(b==0)
        return a;
    return gcd(b,a%b);
}

1. Make

c = g c d (a, b)

$c=gcd(a,b)$

a = m c

$a=mc$

b = n c

$b=nc$

c is the biggest divisor

2. Got

r = a - k b = m c - k n c = (m - k n) c

$r=a-kb=mc-knc=(m-kn)c$

3. `c` is a divisor of `r` as well.

4. `m-kn` and `n` definitely has coprime relation

proof by contradiction

setting they are no coprime relation
$m-kn=xd$
-
$n=yd$ (d>1)
then
$m=kn+xd=kyd+xd=(ky+x)d$
-
$a=mc=(ky+x)dc$
-
$b=nc=ydc$
a and b ‘s biggest divisor is not c. so step 4 proved.

5. `c` is biggest divisor in `b` and `r` as well.

g c d (b, r) = c

$gcd(b, r)=c$

g c d (a, b) = g c d (b, r)

$gcd(a, b)=gcd(b, r)$

RSA Principle 5

Posted on 2017-03-05 | Edited on 2020-06-16 | In math

Proving

Why?

c^{d} \equiv m (m o d n)

$c^d \equiv m \quad (mod \quad n)$

Base on encrypt rule:

m^{e} \equiv c (m o d n)

$m^e \equiv c \quad (mod \quad n)$

c = m^{e} - k n

$c = m^e - kn$

Put it into the target

(m^{e} - k n)^{d} \equiv m (m o d n)

$(m^e - kn)^d \equiv m \quad (mod \quad n)$

The n in left part of equal could all be divided by n. So it’s same as this:

m^{e d} \equiv m (m o d n)

$m^{ed} \equiv m \quad (mod \quad n)$

Bacause

e d \equiv 1 (m o d ϕ (n))

$ed \equiv 1 \quad(mod \quad\phi(n))$

e d = h ϕ (n) + 1

$ed = h\phi(n) + 1$

Put it into above, prove this work

m^{h ϕ (n) + 1} \equiv m (m o d n)

$m^{h\phi(n) + 1} \equiv m \quad (mod \quad n)$

Then divide into two condition

`m` and `n` has coprime relation

In this condition base on euler theorem

m^{ϕ (n)} \equiv 1 (m o d n)

$m^{\phi(n)} \equiv 1 \quad (mod \quad n)$

Put it into above:

(m^{ϕ (n)})^{h} \times m \equiv m (m o d n)

$(m^{\phi(n)})^h \times m \equiv m \quad (mod \quad n)$

Proved

`m` and `n` has no coprime relation

In this condition, bacause m and n has no coprime relation, so m and n must have common factor that is not 1.
n come from two coprime factor p and q, so m must multiply with p or q.
m = kp or m = kq

Choose m = kp for example
Because q is a coprime number, and k has no possible multiply q[or will cross m], so k and q has coprime relation.
Base on euler theorem:

φ (q) = q - 1

$\varphi(q) = q-1$

(k p)^{q - 1} \equiv 1 (m o d q)

$(kp)^{q-1} \equiv 1 \quad (mod \quad q)$

Further

[(k p)^{q - 1}]^{h (q - 1)} \times k p \equiv k p (m o d q)

$[(kp)^{q-1}]^{h(q-1)} \times kp \equiv kp \quad (mod \quad q)$

Bascause

e d = h ϕ (n) + 1

$ed = h\phi(n) + 1$

so:

(k p)^{e d} \equiv k p (m o d q)

$(kp)^{ed} \equiv kp \quad (mod \quad q)$

(k p)^{e d} = k p + t q

$(kp)^{ed} = kp + tq$

This time t definitely could be divided be p. Why?

p ((k p)^{e d - 1} - k) = t q

$p((kp)^{ed-1}-k) = tq$

$((kp)^{ed-1}-k)$ is an integer and p q has coprime relation
So definitely has integer $t’ = t/p$

(k p)^{e d} = k p + t^{'} p q

$(kp)^{ed} = kp + t'pq$

m=kp, n=pq then:

m^{e d} = m (m o d n)

$m^{ed} = m \quad (mod \quad n)$

Proved

RSA Principle 4

Posted on 2017-03-05 | Edited on 2020-06-16 | In math

Encryption and decryption

Pub Key: (14363, 97)
Prv Key: (14363, 9553)

If A want send message m to B.
A need use B’s public key to encrypt M[Integer and M < n].

m^{e} \equiv c (m o d n)

$m^e \equiv c \quad(mod \quad n)$

Suppose A want send “Hello B” to B, the first alphabet ‘H’ should be encrypt like this.

c = 72^{97} m o d 14363

$c = 72^{97} \quad mod \quad 14363$

The ciphertext c is 5769

B received A’s message 5769, B use private key to decrypt it.

c^{d} \equiv m (m o d n)

$c^d \equiv m \quad(mod \quad n)$

M = 5769^{9553} m o d 14363

$M = 5769^{9553} \quad mod \quad 14363$

The text is 72[ ‘H’ in ASCII ]

Signatures and signed inspection certificate

It’s a opposite progress of above one.

Sender in order to show it’s identity, it need to use it’s private key to encrypt[autograph] abstract message like hash code or mac code.

Receiver got message, use public key to decrypt[certificate] the signature, than contract the clear text with message’s abstract.

Terminology

Status

Request Parameters

Create a new index

A new field mapping

Add data

Have a search

Check engine status

Install

Config

Plugin

head

Exception

Operate

Download Configure

Have a try

Contain relation

Equal realtion

Events sum

Events product

Mutex events

Oppose events

Events difference

Complete event group

Proof

Express the average compare

Rearrange terms

PS.

1. Make

2. Got

3. c is a divisor of r as well.

4. m-kn and n definitely has coprime relation

proof by contradiction

5. c is biggest divisor in b and r as well.

Proving

m and n has coprime relation

m and n has no coprime relation

Encryption and decryption

Signatures and signed inspection certificate

3. `c` is a divisor of `r` as well.

4. `m-kn` and `n` definitely has coprime relation

5. `c` is biggest divisor in `b` and `r` as well.

`m` and `n` has coprime relation

`m` and `n` has no coprime relation