Squid with Cache Youtube

Puas rasanya, akhirnya bisa caching youtube (dan yang similar) :D.
Setelah sekian bulan “mangkrak” akhirnya jalan juga walau masih di mesin virtualbox.

Howto ini bukan untuk para pembenci youtube dan googlemap.

Tapi untuk youtube dan googlemap lovers.

bahan referensi yang jadi bacaan.

http://www.mail-archive.com/squid-users@squid-cache.org/msg54605.html
http://www.mail-archive.com/squid-users@squid-cache.org/msg51076.html
http://wiki.squid-cache.org/Features/StoreUrlRewrite
http://wiki.squid-cache.org/Features/StoreUrlRewrite/RewriteScript

Versi yang saya pakai adalah squid-2.7.STABLE3, tidak tahu dukungan untuk versi yang
lain.

1. buat script untuk manipulasi youtube.

2. #!/usr/bin/perl
3. $|=1;
4. while (<>) {
5. @X = split;
6. $url = $X[0];
7. $url
=~s@^http://(.*?)/get_video\?(.*)video_id=(.*?)&.*@squid://videos
.youtube.INTERNAL/ID=$3@;
8. $url
=~s@^http://(.*?)/get_video\?(.*)video_id=(.*?)$@squid://videos.y
outube.INTERNAL/ID=$3@;
9. $url
=~s@^http://(.*?)/videodownload\?(.*)docid=(.*?)$@squid://videos.
google.INTERNAL/ID=$3@;
10.$url
=~s@^http://(.*?)/videodownload\?(.*)docid=(.*?)&.*@squid://video
s.google.INTERNAL/ID=$3@;
print "$url\n"; }

11. Lalu di squid.conf-nya edit seperti yang dibawah ini:

12.acl store_rewrite_list url_regex ^http://(.*?)/get_video\?
13.acl store_rewrite_list url_regex ^http://(.*?)/videodownload\?
14.cache allow store_rewrite_list
15.
16.# Had to uncomment this again, because I couln'd login to google
mail using IE6 (firefox had no trouble):
17.acl QUERY urlpath_regex cgi-bin \?
18.cache deny QUERY
19.
20.refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999
override-expire ignore-no-cache ignore-private
21.refresh_pattern ^http://(.*?)/videodownload\? 10080 90% 999999
override-expire ignore-no-cache ignore-private
22.
 23.storeurl_access allow store_rewrite_list
24.storeurl_access deny all
25.
storeurl_rewrite_program /usr/local/bin/store_url_rewrite

Hasilnya bisa dilihat di access-log, pada saat mengakses video yang sama, akan langsung
hit.

# grep youtube access.log | grep TCP_HIT

1214834411.379 735 192.168.1.89 TCP_HIT/200 1604459 GET

http://youtube.com/get_video?video_id=2d55B-
SiJdM&t=OEgsToPDskKrwAAE_vVIhOqMhPqmPDUQ - NONE/- video/flv
1214834487.090 818 192.168.1.94 TCP_HIT/200 1604459 GET
http://youtube.com/get_video?video_id=2d55B-
SiJdM&t=OEgsToPDskLGVqEnxKjLEN4DGA3HYGse - NONE/- video/flv
1214836269.353 4383 192.168.1.91 TCP_HIT/200 9533167 GET
http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskKeQxYVvYZ
7fgEIW4UNC_U- - NONE/- video/flv
1214836514.802 3757 192.168.1.91 TCP_HIT/200 9533167 GET
http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskIEwsTb26L
iGFc96hBUUa9Z - NONE/- video/flv

Satu pesan dari Horacio Herrera Gonzalez, karena basic scriptnya tidak spesifik ke url
tertentu, maka :

Warning! This code may match other sites not related to YT or GV.

He he he he, watching your bandwidth.

Karena beberapa user merasa kesulitan untuk mengaplied caching youtube.

Langkah dibawah adalah urutan di server saya.

1. Saya pakai distro TSL 3.05, dengan squid squid-2.7.STABLE3

2. ./configure \
3. --sysconfdir=/etc/squid \
4. --prefix=/usr \
5. --enable-async-io \
6. --enable-removal-policies=lru,heap \
7. --disable-delay-pools \
8. --disable-wccp \
9. --disable-wccp2 \
10.--enable-kill-parent-hack \
11.--enable-snmp \
12.--enable-default-err-languages=English --enable-err-
languages=English \
13.--enable-linux-netfilter \
14.--disable-auth
15. config hasil parsing ^# dari squid.conf
16.acl all src all
17.acl manager proto cache_object
18.acl localhost src 127.0.0.1/32
19.acl to_localhost dst 127.0.0.0/8
20.acl localnet src 10.0.0.0/8 # RFC1918 possible internal
network
21.acl localnet src 172.16.0.0/12 # RFC1918 possible internal
network
22.acl localnet src 192.168.0.0/16 # RFC1918 possible internal
network
23.acl SSL_ports port 443
24.acl Safe_ports port 80 # http
25.acl Safe_ports port 21 # ftp
26.acl Safe_ports port 443 # https
27.acl Safe_ports port 70 # gopher
28.acl Safe_ports port 210 # wais
29.acl Safe_ports port 1025-65535 # unregistered ports
30.acl Safe_ports port 280 # http-mgmt
31.acl Safe_ports port 488 # gss-http
32.acl Safe_ports port 591 # filemaker
33.acl Safe_ports port 777 # multiling http
34.acl CONNECT method CONNECT
35.http_access allow manager localhost
36.http_access deny manager
37.http_access deny !Safe_ports
38.http_access deny CONNECT !SSL_ports
39.http_access allow localnet
40.http_access deny all
41.icp_access allow localnet
42.icp_access deny all
43.http_port 3128 transparent
44.hierarchy_stoplist cgi-bin ?
45.cache_mem 6 MB
46.maximum_object_size_in_memory 32 KB
47.memory_replacement_policy heap GDSF
48.cache_replacement_policy heap LFUDA
49.cache_dir aufs /nfs/cache 20000 16 256
50.maximum_object_size 64 MB
51.cache_swap_low 98
52.cache_swap_high 99
53.access_log /var/log/squid/access.log squid
54.cache_log /var/log/squid/cache.log
55.cache_store_log none
56.log_fqdn off
57.storeurl_rewrite_program /etc/squid/store_url_rewrite
58.acl store_rewrite_list url_regex ^http://(.*?)/get_video\?
59.acl store_rewrite_list url_regex ^http://(.*?)/videodownload\?
60.storeurl_access allow store_rewrite_list
61.storeurl_access deny all
62.cache allow store_rewrite_list
63.acl QUERY urlpath_regex cgi-bin \?
64.cache deny QUERY
65.refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999
override-expire ignore-no-cache ignore-private
66.refresh_pattern ^http://(.*?)/videodownload\? 10080 90% 999999
override-expire ignore-no-cache ignore-private
67.refresh_pattern ^ftp: 1440 20% 10080
68.refresh_pattern ^gopher: 1440 0% 1440
69.refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
70.refresh_pattern . 0 20% 4320
 71.quick_abort_min 0
72.quick_abort_max 0
73.quick_abort_pct 98
74.acl apache rep_header Server ^Apache
75.broken_vary_encoding allow apache
76.vary_ignore_expire on
77.cache_effective_user squid
78.cache_effective_group squid
79.log_icp_queries off
80.ipcache_size 2048
81.ipcache_low 98
82.ipcache_high 99
83.memory_pools off
84.reload_into_ims on
85.coredump_dir /usr/var/cache
pipeline_prefetch on

Caching photobucket

Kontribusi apit1, caching untuk photobucket yang banyak di pakai di friendster.

di store_url_rewrite

$url
=~s@^http://(.*?)/albums\?&.*@squid://images.photobucket.INTERNAL/ID=$3
@;
$url
=~s@^http://(.*?)/albums\?$@squid://images.photobucket.INTERNAL/ID=$3@;
$url
=~s@^http://(.*?)/albums\?&.*@squid://videos.photobucket.INTERNAL/ID=$3
@;
$url
=~s@^http://(.*?)/albums\?$@squid://videos.photobucket.INTERNAL/ID=$3@;

di squid.conf

acl store_rewrite_list url_regex

^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\?
acl store_rewrite_list url_regex
^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\?

refresh_pattern
^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\? 43200 90%
999999 override-expire ignore-no-cache ignore-private
refresh_pattern ^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\?
43200 90% 999999 override-expire ignore-no-cache ignore-private

Hasilnya

TCP_HIT/200 5474813 GET

http://vid264.photobucket.com/albums/ii163/shannonwiseman12/DSCN0212.fl
v - NONE/- text/plain
Hasilnya bisa dilihat di access-log, pada saat mengakses video yang sama, akan langsung
hit.

# grep youtube access.log | grep TCP_HIT

1214834411.379 735 192.168.1.89 TCP_HIT/200 1604459 GET

http://youtube.com/get_video?video_id=2d55B-
SiJdM&t=OEgsToPDskKrwAAE_vVIhOqMhPqmPDUQ - NONE/- video/flv
1214834487.090 818 192.168.1.94 TCP_HIT/200 1604459 GET
http://youtube.com/get_video?video_id=2d55B-
SiJdM&t=OEgsToPDskLGVqEnxKjLEN4DGA3HYGse - NONE/- video/flv
1214836269.353 4383 192.168.1.91 TCP_HIT/200 9533167 GET
http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskKeQxYVvYZ
7fgEIW4UNC_U- - NONE/- video/flv
1214836514.802 3757 192.168.1.91 TCP_HIT/200 9533167 GET
http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskIEwsTb26L
iGFc96hBUUa9Z - NONE/- video/flv

Satu pesan dari Horacio Herrera Gonzalez, karena basic scriptnya tidak spesifik ke url
tertentu, maka :

Warning! This code may match other sites not related to YT or GV.

He he he he, watching your bandwidth.

Karena beberapa user merasa kesulitan untuk mengaplied caching youtube.

Langkah dibawah adalah urutan di server saya.

1. Saya pakai distro TSL 3.05, dengan squid squid-2.7.STABLE3

2. ./configure \
3. --sysconfdir=/etc/squid \
4. --prefix=/usr \
5. --enable-async-io \
6. --enable-removal-policies=lru,heap \
7. --disable-delay-pools \
8. --disable-wccp \
9. --disable-wccp2 \
10.--enable-kill-parent-hack \
11.--enable-snmp \
12.--enable-default-err-languages=English --enable-err-
languages=English \
13.--enable-linux-netfilter \
14.--disable-auth
Squid.conf

config hasil parsing ^# dari squid.conf

acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal
network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access deny all
icp_access allow localnet
icp_access deny all
http_port 3128 transparent
hierarchy_stoplist cgi-bin ?
cache_mem 6 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /nfs/cache 20000 16 256
maximum_object_size 64 MB
cache_swap_low 98
cache_swap_high 99
access_log /var/log/squid/access.log squid
cache_log /var/log/squid/cache.log
cache_store_log none
log_fqdn off
storeurl_rewrite_program /etc/squid/store_url_rewrite
acl store_rewrite_list url_regex ^http://(.*?)/get_video\?
acl store_rewrite_list url_regex ^http://(.*?)/videodownload\?
storeurl_access allow store_rewrite_list
storeurl_access deny all
cache allow store_rewrite_list
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999 override-
expire ignore-no-cache ignore-private
refresh_pattern ^http://(.*?)/videodownload\? 10080 90% 999999
override-expire ignore-no-cache ignore-private
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
quick_abort_min 0
quick_abort_max 0
quick_abort_pct 98
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
vary_ignore_expire on
cache_effective_user squid
cache_effective_group squid
log_icp_queries off
ipcache_size 2048
ipcache_low 98
ipcache_high 99
memory_pools off
reload_into_ims on
coredump_dir /usr/var/cache
pipeline_prefetch on

Caching photobucket

Kontribusi apit1, caching untuk photobucket yang banyak di pakai di friendster.

di store_url_rewrite

$url
=~s@^http://(.*?)/albums\?&.*@squid://images.photobucket.INTERNAL/ID=$3
@;
$url
=~s@^http://(.*?)/albums\?$@squid://images.photobucket.INTERNAL/ID=$3@;
$url
=~s@^http://(.*?)/albums\?&.*@squid://videos.photobucket.INTERNAL/ID=$3
@;
$url
=~s@^http://(.*?)/albums\?$@squid://videos.photobucket.INTERNAL/ID=$3@;

di squid.conf

acl store_rewrite_list url_regex

^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\?
acl store_rewrite_list url_regex
^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\?

refresh_pattern
^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\? 43200 90%
999999 override-expire ignore-no-cache ignore-private
refresh_pattern ^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\?
43200 90% 999999 override-expire ignore-no-cache ignore-private

Hasilnya

TCP_HIT/200 5474813 GET

http://vid264.photobucket.com/albums/ii163/shannonwiseman12/DSCN0212.fl
v - NONE/- text/plain