A1 , A 2 ili ili A3 B1 , B 2 , B 3 , B 4 iliB5 A1 , A 2 i A3
B j , j = 1, 2, 2 , 3, 3 , 4, 4 , 5 Ai , i = 1, 2, 2 , 3
p(B p(B j |Ai ) A1 A2 A3
B1
B2
B3
B4
B5
3
H(X) = − = −
0.1 − 0 0.6 − 0 0.3 p(A p(Ai )log2 p(A p(Ai ) = −0 − 0.1log2 0. − 0..6log2 0. − 0..3log2 0.
i=1
≈ 1. 1 .3
5
H(Y) = − = −
p(B p(B j )log2 p(B p(B j )
j =1
3
p(B p(B j ) =
p(A p(Ai ) p(B p(B j |Ai )
i=1
p(B p(B1 ) = 0.1 · 0 · 0..3 + 0. 0 .6 · 0 · 0..35 + 0. 0.3 · 0 · 0..15 = 0. 0.285 p(B p(B2 ) = 0.1 · 0 · 0..1 + 0. 0 .6 · 0 · 0..3 + 0. 0 .3 · 0 · 0..4 = 0.31 p(B p(B3 ) = 0.1 · 0 · 0..05 + 0. 0.6 · 0 · 0..15 + 0. 0.3 · 0 · 0..2 = 0.155 · 0..15 + 0. 0.6 · 0 · 0..05 + 0. 0.3 · 0 · 0..05 = 0. 0.06 p(B p(B4 ) = 0.1 · 0 · 0..4 + 0. 0 .6 · 0 · 0..15 + 0. 0.3 · 0 · 0..2 = 0.19 p(B p(B5 ) = 0.1 · 0 5
H(Y) = − = −
p(B p(B j )log2 p(B p(B j )
j =1
= −0 − 0.285log2 0. 0.285 − 285 − 0 0..31log2 0. 0.31 − 31 − 0 0..155log2 0. 0.155 − 155 − 0 0..06log2 0. 0.06 − 0. 0.19log2 0. 0.19 ≈ 2. 2 .156 5
H(X, H(X, Y) = −
3
p(B p(B j ) p(A p(Ai |B j )log2 p(A p(Ai |B j )
j =1 i=1 5
= −
3
p(B p(B j |Ai ) p(A p(Ai )log 2 p(B p(B j |Ai ) p(A p(Ai )
j =1 i=1
≈ 3. 3 .375 H(X| H(X|Y) = H(X, H(X, Y) − Y) − H(Y) H(Y) ≈ ≈ 3 3..375 − 375 − 2 2..156 ≈ 156 ≈ 1 1..22 H(Y| H(Y|X) = H(X, H(X, Y) − Y) − H(X) H(X) ≈ ≈ 3 3..375 − 375 − 1 1..3 = 2. 2 .075
I(X, Y) = H(X) + H(Y) − H(X, Y) ≈ 1.3 + 2.156 − 3.375 ≈ 0.081
p(X) = 0.18 p(Y|X) = 0.41, p(Y|X) = 0.46 p(Z|Y) = 1, p(Z|YX) = 0.27 i p(Z|Y X) = 0.56
p(X) = 1 − p(X) = 1 − 0.18 = 0.82 p(Y) = p(X) p(Y|X) + p(X) p(Y|X) = 0.82 · 0.46 + 0.18 · 0.41 ≈ 0.451 p(Y) = 1 − p(Y) = 1 − 0.451 = 0.549 p(Z|X) = p(Y) p(Z|YX) + p(Y) p(Z|YX) = 0.549 · 0.27 + 0.451 · 1 ≈ 0.6 p(Z|X) = 1 − p(Z|X) = 1 − 0.6 = 0.4 p(Z|X) = p(Y) p(Z|Y X) + p(Y) p(Z|YX) = 0.549 · 0.56 + 0.451 · 1 ≈ 0.76 p(Z|X) = 1 − p(Z|X) = 1 − 0.76 = 0.24 p(Z) = p(X) p(Z|X) + p(X) p(Z|X) = 0.82 · 0.76 + 0.18 · 0.6 ≈ 0.73 p(Z) = 1 − p(Z) = 1 − 0.73 = 0.27
I(X, Z) = − p(X) p(Z|X)log2
p(Z|X) p(Z|X) − p(X) p(Z|X)log2 p(Z) p(Z)
p(Z|X) p(Z|X) − p(X) p(Z|X)log2 p(Z) p(Z) 0.6 0.4 = −0.18 · 0.6log2 − 0.18 · 0.4log2 0.73 0.27 0.76 0.24 − 0.82 · 0.76log2 − 0.82 · 0.24log2 0.73 0.27 = −0.013 − p(X) p(Z|X)log2
a b c
p(x j |Si ) Sa Sb Sc Sd
a
b
c
a b c
d
a
b
c
d
d
d
bcbaddca
0.1/b
b
0.1/a
0.45/b 0.3/b
0.35/d 0.1/a
0.05/a
0.45/c
a
0.3/b
d
0.3/d 0.25/a
0.05/c
0.35/c
0.1/d
c
0.35/c p( i )
0.4/d
p(Sa ) =
p(Si ) p(a|Si )
i ∈ {a,b,c,d}
= 0.05 p(Sa ) + 0.1 p(Sb ) + 0.25 p(Sc ) + 0.1 p(Sd ) p(Sb ) = 0.3 p(Sa ) + 0.1 p(Sb ) + 0.3 p(Sc ) + 0.45 p(Sd ) p(Sc ) = 0.35 p(Sa ) + 0.45 p(Sb ) + 0.35 p(Sc ) + 0.05 p(Sd ) p(Sa ) + p(Sb ) + p(Sc ) + p(Sd ) = 1 p(Sa ) = 0 .1371 p(Sb ) = 0.2855 p(Sc ) = 0.2933
H(Sa ) = −
p(Sd ) = 0.284
p(i|Sa )log 2 p(i|Sa )
i ∈ {a,b,c,d}
= −0.05log2 0.05 − 0.3log2 0.3 − 0.35log2 0.35 − 0.3log2 0.3 ≈ 1.7884 H(Sb ) = −0.1log2 0.1 − 0.1log2 0.1 − 0.45log2 0.45 − 0.35log2 0.35 ≈ 1.7129 H(Sc ) = −0.25log2 0.25 − 0.3log2 0.3 − 0.35log2 0.35 − 0.1log2 0.1 ≈ 1.8834 H(Sd ) = −0.1log2 0.1 − 0.45 log2 0.45 − 0.05log2 0.05 − 0.4log2 0.4 ≈ 1.5955
H(X|X∞ ) =
p(Si )H(Si )
i ∈ {a,b,c,d}
= 0.1371 · 1.7884 + 0.2855 · 1.7129 + 0.2933 · 1.8834 + 0.284 · 1.5955 ≈ 1.7397
R =
log2 4 − H(X|X∞ ) 2 − 1.7397 ≈ ≈ 0.1301 ≈ 13% log2 4 2
H(X5 ) = H(X1 ) + (5 − 1)H(X|X∞ ) H(X1 ) = −
p(i)log2 p(i)
i ∈ {a,b,c,d}
= −0.1371 log2 0.1371 − 0.2855 log2 0.2855 − 0.2933 log2 0.2933 − 0.284 log2 0.284 ≈ 1.9442 H(X5 ) = H(X1 ) + 4H(X|X∞ ) ≈ 1.9442 + 4 · 1.7397 ≈ 8.9030 bcbaddca p(bcbaddca) = p(b) p(c|b) p(b|c) p(a|b) p(d|a) p(d|d) p(c|d) p(a|c) = 0.2855 · 0.45 · 0.3 · 0.1 · 0.3 · 0.4 · 0.05 · 0.25 ≈ 5.78 · 10−4 %
00
01
10
11 01
p(0|S00 ) = 0.3 p(0|S01 ) = 0.8 p(0|S10 ) = 0.5 p(0|S11 ) = 0.2
01
0.7/1
0.3/0
0.2/1
0.8/0
00
0.5/1
0.5/0
11
0.2/0
10
p( i )
i = 1, 2, 3, 4
p(S00 ) = p(S00 ) p(0|S00 ) + p(S10 ) p(0|S10 ) = 0.3 p(S00 ) + 0.5 p(S10 ) p(S01 ) = p(S00 ) p(1|S00 ) + p(S10 ) p(1|S10 ) = 0.7 p(S00 ) + 0.5 p(S10 ) p(S10 ) = p(S01 ) p(0|S01 ) + p(S11 ) p(0|S11 ) = 0.8 p(S01 ) + 0.2 p(S11 ) p(S00 ) + p(S01 ) + p(S10 ) + p(S11 ) = 1 p(S00 ) = 0.1923 p(S01 ) = 0.2692 p(S10 ) = 0.2692
p(S11 ) = 0.2692
0.8/1
H(S00 ) = −
p(i|S00 )log2 p(i|S00 )
i ∈ {0, 1}
= −0.3log2 0.3 − 0.7log2 0.7 ≈ 0.8813 H(S01 ) = −0.8log2 0.8 − 0.2log2 0.2 ≈ 0.7219 H(S10 ) = −0.5log2 0.5 − 0.5log2 0.5 = 1 H(S11 ) = −0.2log2 0.2 − 0.8log2 0.8 ≈ 0.7219
H(X|X∞ ) =
p(Si )H(Si )
i ∈ {00, 01, 10, 11}
= 0.1923 · 0.8813 + 0.2692 · 0.7219 + 0.2692 · 1 + 0.284 · 0.7219 ≈ 0.8380
R =
log2 2 − H(X|X∞ ) ≈ 1 − 0.8380 = 0.162 = 16.2% log2 2
H(X8 ) = H(X) + H(X|X1 ) + (8 − 2)H(X|X∞ ) H(X) = −
p(i)log 2 p(i)
i ∈ {0, 1}
p(0) =
p(0|Si ) p(Si ) i ∈ {00, 01, 10, 11} = 0.3 · 0.1923 + 0.8 · 0.2692 + 0.5 · 0.2692 + 0.2 · 0.2692 ≈ 0.4616
p(1) = 1 − p(0) ≈ 0.5384 H(X) = −0.4616 log2 0.4616 − 0.5384 log2 0.5384 ≈ 0.9957 H(X|X1 ) = −
p(i)
p( j|i)log 2 p( j|i)
i ∈ {0, 1} j ∈ {0, 1}
p(0|S00 ) p(S00 ) + p(0|S10 ) p(S10 ) p(S00 ) + p(S10 ) 0.3 · 0.1923 + 0.5 · 0.2692 = ≈ 0.4167 0.1923 + 0.2692
p(0|0) =
p(0|S01 ) p(S01 ) + p(0|S11 ) p(S11 ) p(S01 ) + p(S11 ) 0.8 · 0.2692 + 0.2 · 0.2692 = ≈ 0.5 0.2692 + 0.2692 p(1|0) = 1 − p(0|0) ≈ 1 − 0.4167 = 0.5833 p(0|1) =
p(1|1) = 1 − p(0|1) ≈ 1 − 0.5 = 0.5 H(X|X1 ) = − p(0)( p(0|0)log2 p(0|0) + p(1|0)log2 p(1|0)) − p(1)( p(0|1)log2 p(0|1) + p(1|1)log2 p(1|1)) ≈ −0.4616 (0.4167 log2 0.4167 + 0.5833 log2 0.5833) −0.5384 (0.5log2 0.5 + 0.5log2 0.5) = 0.9907 H(X8 ) = H(X) + H(X|X1 ) + 6H(X|X∞ ) ≈ 0.9957 + 0.9907 + 6 · 0.838 ≈ 7.0144
p(101101) = p(S10 ) p(1|S10 ) p(1|S01 ) p(0|S11 ) p(1|S10 ) = 0.2962 · 0.5 · 0.2 · 0.2 · 0.5 ≈ 0.0027 = 0.27%
→ →
→
10
nsr =
pi ni ≈
i=1
1 N
→
→
→
→
→
→
10
N i ni
i=1
1 = (37 · 4 + 11 · 5 + 52 · 3 + 97 · 2 + 93 · 3 + 65 · 3 + 21 · 5 + 45 · 4 + 35 · 4 + 66 · 3) 522 ≈ 3.1609
10
H(X|X∞ ) = −
pi log2 pi ≈ log 2 N −
i=1
1 N
10
N i log2 N i
i=1
1 = log2 522 − (37log2 37 + 11 log2 11 + 52 log2 52 + 97log2 97 + +93 log2 93 + 65 log2 65 522 + 21 log2 21 + 45 log2 45 + 35 log2 35 + 66 log2 66) ≈ 3.11823
I(X) =
H(X|X∞ ) 3.11825 0.9865 ≈ ≈ nsr τ 3.1609τ τ Cc = log2 2/τ = 1/τ
→
→
→
10
nsr =
pi ni ≈
i=1
→ →
→
1 N
→
→
→
→
10
N i ni
i=1
1 = (37 · 4 + 11 · 5 + 52 · 3 + 97 · 2 + 93 · 3 + 65 · 3 + 21 · 5 + 45 · 4 + 35 · 4 + 66 · 3) 522 ≈ 3.1609
10
H(X|X∞ ) = −
pi log2 pi ≈ log 2 N −
i=1
1 N
10
N i log2 N i
i=1
1 = log2 522 − (37log2 37 + 11 log2 11 + 52 log2 52 + 97log2 97 + +93 log2 93 + 65 log2 65 522 + 21 log2 21 + 45 log2 45 + 35 log2 35 + 66 log2 66) ≈ 3.11823
I(X) =
H(X|X∞ ) 3.11825 0.9865 ≈ ≈ nsr τ 3.1609τ τ Cc = log2 2/τ = 1/τ
m = 3
n = 10
m∗ = 2 + mod (10 − 4, 3 − 1) = 2 + mod (6, 2) = 2
→
→ →
→
10
nsr =
→
→
→
→
→
→
→
1 pi ni ≈ N
i=1
10
N i ni
i=1
1 (37 · 2 + 11 · 3 + 52 · 2 + 97 · 2 + 93 · 2 + 65 · 2 + 21 · 3 + 45 · 2 + 35 · 2 + 66 · 2) 522 ≈ 2.0613 =
10
∞
H(X|X ) = −
1 pi log2 pi ≈ log 2 N − N
i=1
10
N i log2 N i
i=1
1 (37log2 37 + 11 log2 11 + 52 log2 52 + 97log2 97 + +93 log2 93 + 65 log2 65 522 + 21 log2 21 + 45 log2 45 + 35 log2 35 + 66 log2 66)
= log2 522 −
≈ 3.11823
I(X) =
H(X|X∞ ) 3.11825 1.5128 ≈ ≈ nsr τ 2.0613τ τ Cc = log2 3/τ ≈ 1.585/τ
p(A) = 0.05 p(B) = 0.2 p(C) = 0.4 p(D) = 0.35
→
→
→
→
4
nsr =
pi ni
i=1
= 0.05 · 3 + 0.2 · 3 + 0.4 · 1 + 0.35 · 2 ≈ 1.85
4
∞
H(X|X ) = −
pi log2 pi
i=1
= 0.05log2 0.05 + 0.2log2 0.2 + 0.4log2 0.4 + 0.35log2 0.35 ≈ 1.7394
I(X) =
H(X|X∞ ) 1.7394 0.94 ≈ ≈ nsr τ 1.85τ τ Cc = log2 2/τ = 1/τ
→
4
nsr =
pi ni
i=1
= 0.05 · 3 + 0.2 · 3 + 0.4 · 1 + 0.35 · 2 ≈ 1.85
→
→
→
4
∞
H(X|X ) = −
pi log2 pi
i=1
= 0.05log2 0.05 + 0.2log2 0.2 + 0.4log2 0.4 + 0.35log2 0.35 ≈ 1.7394
I(X) =
H(X|X∞ ) 1.7394 0.94 ≈ ≈ 1.85τ nsr τ τ Cc = log2 2/τ = 1/τ
→ →
→ →
16
nsr =
1 pi ni ≈ N
i=1
→
→ →
→ → →
→
→ →
→ →
→
10
N i ni
i=1
1 = (1 · 8 + 4 · 7 + 8 · 5 + 7 · 6 + 4 · 8 + 16 · 5 + 32 · 4 + 28 · 4 + 400 8 · 6 + 32 · 4 + 64 · 2 + 56 · 3 + 7 · 6 + 28 · 4 + 56 · 3 + 49 · 3) ≈ 3.5275 H(X|X∞ ) = H(X2 ) 3.4788
H(X2 ) = 2H(X)
H(X|X∞ ) = 2H(X) = 2 · 1.7394 ≈
I(X) =
H(X|X∞ ) 3.4788 0.9862 ≈ ≈ 3.5275τ nsr τ τ Cc = log2 2/τ ≈ 1/τ
