확률변수,RV

random variable






1. 먼저

초보자에게 헷갈리는 것
  • 확률변수는 사실 변수가 아니고 함수다.
    함수이지만 변수처럼 다룬다고 해서 이름이 확률변수인가?
  • 확률변수는 보통 알파벳 X를 쓴다. 이름이 변수인데다 문자도 X로 쓰고 소문자 x도 같이 나오니 혼란스럽다.
  • 대문자 X, Y, Z 등은 확률변수(즉 함수)고, 그것이 가질 수 있는 값?(치역의 원소? 정의역의 원소? 무엇?) CHK 을 소문자 x, y, z등으로 쓰는 것 같다.

2. Definition of RV

A RV $\displaystyle X$ is a real-valued function of the experimental outcome.
$\displaystyle X:\mathbb{\Omega}\to\mathbb{R}$
여기서
$\displaystyle \mathbb{\Omega}$ = VG:표본공간,sample_space

3. Discrete RV (DRV)

A RV $\displaystyle X$ is discrete if its range is finite or countably infinite.
여기서, range $\displaystyle r(X)=\left{x\middle|\exists\omega\in\mathbb{\Omega}\textrm{ such that }X(\omega)=x\right}$
즉 sample $\displaystyle \omega$$\displaystyle x$ 라는 값으로 대응시키는?
그런 함수가 $\displaystyle X$ 이고 그것을 독립변수(or 정의역)로 하는 대응관계가 range r? 조건제시법 이해가 잘...

ex. two fair coin tosses
X = # of heads
Ω = {HH, HT, TH, TT}
X(HH) = 2
X(TT) = 0
r(X) = {0, 1, 2}

ex. sampling a number ω from [-1,1]
$\displaystyle X(\omega)=\begin{cases}1,&\textrm{ if }\omega>0\\0,&\textrm{ if }\omega=0\\-1,&\textrm{ if }\omega<0\end{cases}$


4. Probability mass function (PMF)

The PMF $\displaystyle p_X(x)$ of a DRV $\displaystyle X$ is defined as:
$\displaystyle p_X(x)=P(X=x)=P\left(\left{\omega\in\mathbb{\Omega}\middle|X(\omega)=x\right}\right)$

위의 동전던지기를 예로 들면
$\displaystyle p_X(x)=$
¼ if x=0 TT
½ if x=1 HT TH
¼ if x=2 HH

5. Mean or expectation

평균 or 기대값
$\displaystyle E[X]=\sum_x x\cdot P_X(x)$

위 동전던지기를 예로 들면 앞면이 나오는 횟수의 기대값은
E[X]= 0·¼ + 1·½ + 2·¼ = 1

6. Variance

분산
$\displaystyle V[X]=E[(X-E[X])^2]=\sum_x(x-E[X])^2P_X(x)$

Properties 특징
i) $\displaystyle E[aX+b]=aE[X]+b$
ii) $\displaystyle {\rm Var}[aX+b]=a^2{\rm Var}(X)$

7. Conditioning RV on event

Given an event $\displaystyle A$ with $\displaystyle P(A)>0,$ the conditional PMF $\displaystyle P_{X|A}$ of a DRV $\displaystyle X$ is defined as
$\displaystyle P_{X|A}(x)=P(X=x|A)=\frac{P(\left{X=x\right}\cap A)}{P(A)}$

Conditioning X on Y:
$\displaystyle P_{X|Y}(x|y)=P(X=x|Y=y)$
$\displaystyle =\frac{P(X=x,Y=y)}{P(Y=y)}$
$\displaystyle =\frac{P_{X,Y}(x,y)}{P_Y(y)}$

$\displaystyle =P(\{X=x\}\cap\{Y=y\})$
$\displaystyle =P(\{\omega\in\mathbb{\Omega}|X(\omega)=x\textrm{ and }Y(\omega)=y\})$

8. Conditional expectation

$\displaystyle E[X|A]=\sum_x x\cdot P_{X|A}(x)$

9. Joint PMF of two DRVs

X, Y: DRVs
$\displaystyle P_{X,Y}(x,y)=P(X=x,Y=y)$

10. Independence

Two DRVs $\displaystyle X$ and $\displaystyle Y$ are independent if
$\displaystyle P_{X,Y}(x,y)=P_X(x)P_Y(y)\;\;\;\forall x,y$

11. Continuous RV (CRV)

A RV $\displaystyle X$ is continuous if there exists a non-negative function $\displaystyle f_X()$ such that
$\displaystyle P(X\in B)=\int_B f_X(x)dx,\;\;\forall B\subset\mathbb{R}$
interval B가 $\displaystyle B=[a,b]$ 라면
$\displaystyle P(a\le X\le b)=\int_a^b f_X(x)dx$

페이지: 연속확률변수,continuous_RV

12. DRV and CRV

DRV CRV
PF PMF
$\displaystyle P_X(x)=P(X=x)$
PDF
$\displaystyle f_X(x),P(X\in B)=\int_B f_X(x)dx$
DF CDF
$\displaystyle F_X(k)=P(X\le k)=\sum_{x\le k} P_X(x)$
CDF
$\displaystyle F_X(x)=P(X\le x)=\int_{-\infty}^x f_X(t)dt$
Mean $\displaystyle E[X]=\sum_x xP_X(x)$ $\displaystyle E[X]=\int_{-\infty}^{\infty} xf_X(x)dx$
Var $\displaystyle V(X)=\sum(x-E[X])^2P_X(x)$ $\displaystyle V(X)=\int_{-\infty}^{\infty}(x-E[X])^2dx$

13. 독립 and Joint

discrete에 independence가 있다면
continuous에 joint가 있다?

raised comma=⸴

Independence PX⸴Y(x,y)=PX(x)·PY(y) fX⸴Y(x,y)=fX(x)·fY(y), ∀x,y
Conditioning PX|Y(x|y)=PX⸴Y(x,y) / PY(y) fX|Y(x|y)=fX⸴Y(x,y) / fY(y)

이제부터 $\displaystyle P$$\displaystyle \mathbb{P}$ 를 구분하겠음

14. Total expectation theorem

A1, … An: partition of Ω
$\displaystyle \mathbb{P}(A_i)>0$

$\displaystyle E[X]=\sum_{i=1}^n \mathbb{P}(A_i)E[X|A_i]$
$\displaystyle =\sum_{i} \mathbb{P}(A_i) \sum_x x P_{X|A_i}(x)$
$\displaystyle =\sum_{i} \mathbb{P}(A_i) \sum_x x\cdot \mathbb{P}(X=x|A_i)$
$\displaystyle =\sum_x x \sum_i \mathbb{P}(A_i) \mathbb{P}(X=x|A_i)$
전확률정리에 의해
$\displaystyle =\sum_x x \mathbb{P}(X=x)$

15. Markov inequality

대충 RV가 얼마 이상일 확률의 상한을 제시하는 그런 것인듯
그 상한은 평균에 비례.

Let $\displaystyle X$ be a nonnegative RV. Then $\displaystyle \forall a>0,$
$\displaystyle \mathbb{P}(X\ge a)\le \frac{E[X]}a$

Proof)
$\displaystyle 1_{\{x\ge a\}}=\begin{cases}1,&\text{ if }x\ge a\\0,&\text{ if }x\lt a\end{cases}$
여기서 1은 indicator function
이하 첨자의 중괄호를 ()로 쓰겠음 (입력하기 힘들어서)

$\displaystyle a\cdot1_{(x\ge a)} \le X$
$\displaystyle \Rightarrow E[a\cdot1_{(x\ge a)}]\le E[X]$
좌변 $\displaystyle =a\cdot\mathbb{P}(x\ge a)+0\cdot \mathbb{P}(x\lt a)$
$\displaystyle =a\cdot \mathbb{P}(x\ge a)$

Ex. uniform random variable X
$\displaystyle X\sim{\rm Unif}[0,10]$
$\displaystyle f_X(x)=\begin{cases}1/10,&\text{ if }0\le x\le 10\\0,&\text{otherwise}\end{cases}$
$\displaystyle E(X)=5$
$\displaystyle a=5$ 를 가정
$\displaystyle \mathbb{P}(X\ge 5)\le \frac55 = 1$
$\displaystyle \mathbb{P}(X\ge 6)\le\frac56$

(from 이향원 건대강의)


Def.
Let RV $\displaystyle X(\ge 0)$ with $\displaystyle E(X)$ and 상수 $\displaystyle a(>0)$ then,
$\displaystyle P(X\ge a)\le\frac{E(X)}{a}$

왜냐하면, 연속확률변수(c.r.v.) X에 대해,
$\displaystyle E(X)=\int_0^{\infty}xf_X(x)dx$
$\displaystyle =\int_0^a xf_X(x)dx+\int_a^{\infty}xf_X(x)dx$
$\displaystyle \ge\int_a^{\infty}xf_X(x)dx$
그런데 $\displaystyle a\le x<\infty$ 이므로,
$\displaystyle \ge\int_a^{\infty}af_X(x)dx$
$\displaystyle =a\int_a^{\infty}f_X(x)dx$
$\displaystyle =aP(X\ge a)$
따라서
$\displaystyle P(X\ge a)\le\frac{E(X)}{a}$

from 경북대 최영숙 확률과정 13. The Markov and Chebyshev Inequalities Multiple Random Variables http://www.kocw.net/home/search/kemView.do?kemId=1279832

16. Chebyshev inequality

체비셰프 부등식

For a RV $\displaystyle X$ with mean $\displaystyle \mu$ and variance $\displaystyle \sigma^2,\forall a>0,$
$\displaystyle \mathbb{P}(|X-\mu|\ge a)\le\frac{\sigma^2}{a^2}$

좌변을 다르게 표현하면
$\displaystyle \mathbb{P}(X-\mu>a\textrm{ or }X-\mu\le-a)$
아울러 다음도 당연히 성립
$\displaystyle \mathbb{P}(X-\mu\ge a)\le \mathbb{P}(X-\mu>a\textrm{ or }X-\mu\le-a)$

see also VG:부등식,inequality

Proof)
$\displaystyle \mathbb{P}(|X-\mu|\ge a)=\mathbb{P}((X-\mu)^2\ge a^2)$
Markov ineq.를 적용하면
$\displaystyle =\frac{E[(X-\mu)^2]}{a^2}$

from 건국대


Def.
Let $\displaystyle E(X)=m$ and $\displaystyle Var(X)=\sigma^2$ of R.V. $\displaystyle X.$
then, $\displaystyle P(|X-m|\ge a)\le\frac{\sigma^2}{a^2}\;(a>0)$

이유:
Let $\displaystyle D^2=(X-m)^2.$
By M.I.(Markov ineq.),
$\displaystyle P(D^2\ge a^2)\le\frac{E(D^2)}{a^2}=\frac{\sigma^2}{a^2}$

위 좌변의 괄호 안의 $\displaystyle D^2\ge a^2$ 은,
$\displaystyle \Leftrightarrow D\ge a,D\le -a$
$\displaystyle \Leftrightarrow |D|\ge a$
$\displaystyle \Leftrightarrow |X-m|\ge a$

Note 1.
$\displaystyle P(|X-m|
$\displaystyle \ge1-\frac{\sigma^2}{a^2}$ (= lower bound = 하한)

Note 2.
$\displaystyle P(X=m)=1$ for $\displaystyle \sigma^2=0.$
because, by Note 1, $\displaystyle P(|X-m|
$\displaystyle P(|X-m| (???)
In CI(Chebyshev ineq.), consider $\displaystyle a=k\sigma(>0).$
$\displaystyle P(|X-m|\ge k\sigma)\le\frac{\sigma^2}{(k\sigma)^2}=\frac1{k^2}.$

좌변 괄호 안을 생각하면
For Gaussian random variable $\displaystyle X,$
$\displaystyle \left|\frac{X-m}{\sigma}\right|\ge k$
$\displaystyle Z:$ standard Gaussian random variable.

ex.
If continuous RV X with E(X)=15 and σ=3, find the upper bound for P(|X-m|>5).
sol.
By Chebyshev ineq, $\displaystyle P(|X-m|>5)\le\frac9{25}=0.36$

from 경북대 최영숙 확률과정 13. The Markov and Chebyshev Inequalities Multiple Random Variables http://www.kocw.net/home/search/kemView.do?kemId=1279832

17. Chernoff bound

Let $\displaystyle X_1,\cdots,X_n$ be i.i.d. Bernoulli RVs with parameter $\displaystyle p.$
Let
$\displaystyle S_n=\sum_{i=1}^n X_i,$
$\displaystyle \mu=E[S_n]=np.$
Then,
$\displaystyle \mathbb{P}(S_n\ge(1+\varepsilon)\mu)\le\left(\frac{e^{-\varepsilon}}{1+\varepsilon}\right)^{1+\varepsilon},\;\;\;\;\forall\varepsilon>0$

$\displaystyle \mathbb{P}$ 안의 내용
$\displaystyle S_n\ge(1+\varepsilon)\mu$ 을 다시 쓰면
$\displaystyle S_n-\mu\ge\varepsilon\mu$
즉, 평균에서 $\displaystyle \varepsilon\mu$ 만큼 벗어날 확률이 오른쪽 식만큼 bound가 된다.

또 다른 theorem:
$\displaystyle \mathbb{P}(S_n\le(1-\varepsilon)\mu)\le\left(\frac{e^{-\varepsilon}}{1-\varepsilon}\right)^{1-\varepsilon},\quad\quad 0<\varepsilon<1.$

증명에 앞서
(independent하면)
$\displaystyle E[XY]=E[X]E[Y]$
일반적으로
$\displaystyle E\left[\prod_i X_i\right]=\prod_i E[X_i]$
곱하기의 평균 = 평균의 곱하기

Proof)
$\displaystyle \mathbb{P}(S_n\ge(1+\epsilon)\mu)$
$\displaystyle =\mathbb{P}(tS_n\ge(1+\epsilon)\mu t)$
$\displaystyle =\mathbb{P}(e^{tS_n}\ge e^{(1+\epsilon)\mu t})$ Markov ineq.를 적용하면,
$\displaystyle \le\frac{E[e^{tS_n}]}{e^{(1+\epsilon)\mu t}}$
우변을 풀어서 써보면
$\displaystyle =e^{-(1+\epsilon)\mu t}E\left[\prod_{i=1}^n e^{tX_i}\right]$ 위의 independent 성질을 이용하면
$\displaystyle =e^{-(1+\epsilon)\mu t}\prod_{i=1}^n E[e^{tX_i}]$ (∵ independence of Xi's)
$\displaystyle =e^{-(1+\epsilon)\mu t}\prod_{i=1}^n\left( p\cdot e^t + (1-p) \right)$
$\displaystyle =e^{-(1+\epsilon)\mu t}\prod_{i=1}^n\left( 1+(e^t-1)p \right)$
부등식 $\displaystyle 1+x\le e^x$ 를 이용하면
$\displaystyle \le e^{-(1+\epsilon)\mu t}\prod_{i=1}^n e^{(e^t-1)p}$
$\displaystyle =e^{-(1+\epsilon)\mu t}e^{(e^t-1)\mu}$
$\displaystyle =e^{-(1+\epsilon)\mu t+e^t\mu-\mu$
이 값을 최소화하려면, upper bound가 작으면 작을수록 좋으니까, t에 대해 미분해서 값이 0이 될 때를 생각하면
minimized at $\displaystyle t=\log(1+\epsilon)$
$\displaystyle =e^{(-(1+\epsilon)\log(1+\epsilon)+1+\epsilon-1)\mu}$
$\displaystyle =e^{(\log(1+\epsilon)^{-(1+\epsilon)}+\epsilon)\mu}$
$\displaystyle =e^{\mu\log((1+\epsilon)^{-(1+\epsilon)}e^{\epsilon})$
$\displaystyle =\left(\frac{e^{\epsilon}}{(1+\epsilon)^{1+\epsilon}}\right)^{\mu}$

$\displaystyle \mathbb{P}(S_n\ge(1+\epsilon)\mu)\le\left(\frac{e^{ \epsilon}}{(1+\epsilon)^{1+\epsilon}}\right)^{\mu}\le e^{-\frac{\epsilon^2\mu}{3}}$
$\displaystyle \mathbb{P}(S_n\ge(1-\epsilon)\mu)\le\left(\frac{e^{-\epsilon}}{(1-\epsilon)^{1-\epsilon}}\right)^{\mu}\le e^{-\frac{\epsilon^2\mu}{2}}$

$\displaystyle (1-\epsilon)^{1-\epsilon}>\exp(-\epsilon+\frac12\epsilon^2)$
$\displaystyle (1-\epsilon)\log(1-\epsilon)>-\epsilon+\frac12\epsilon^2$

(from 이향원 건대강의)



(from 이향원 [http]건대강의)