You are on page 1of 9

8/9/2016

TheNetflixTechBlog:ProtectingNetflixViewingPrivacyatScale
4

More NextBlog

CreateBlog SignIn

Monday,August8,2016

Links

ProtectingNetflixViewingPrivacyatScale

NetflixUS&CanadaBlog
NetflixAmericaLatinaBlog

OntheOpenConnectteamatNetflix,wearealwaysworkingtoenhancethehardwareand
softwareinthepurposebuiltOpenConnectAppliances(OCAs)thatstoreandserveNetflixvideo
content.Aswementionedinarecentcompanyblogpost,sincethebeginningoftheOpen
ConnectprogramwehavesignificantlyincreasedtheefficiencyofourOCAsfromdelivering8
Gbpsofthroughputfromasingleserverin2012toover90Gbpsfromasingleserverin2016.We
contributetothiseffortonthesoftwaresidebyoptimizingeveryaspectofthesoftwareforour
uniqueusecaseinparticular,focusingontheopensourceFreeBSDoperatingsystemandthe
NGINXwebserverthatrunontheOCAs.

NetflixBrasilBlog
NetflixBeneluxBlog
NetflixDACHBlog
NetflixFranceBlog
NetflixNordicsBlog
NetflixUK&IrelandBlog
NetflixISPSpeedIndex

MembersoftheteamwillbepresentingatechnicalsessiononthistopicattheIntelDeveloper
Forum(IDF16)inSanFranciscothismonth.Thisblogintroducessomeoftheworkwevedone.

OpenpositionsatNetflix
NetflixWebsite

AddingTLStoVideoStreams

FacebookNetflixPage
NetflixUIEngineering

Inthemoderninternetworld,wehavetofocusnotonlyonefficiency,butalsosecurity.Thereare
manystateoftheartsecuritymechanismsinplaceatNetflix,includingTransportLevelSecurity
(TLS)encryptionofcustomerinformation,searchqueries,andotherconfidentialdata.Wehave
alwaysreliedonpreencodedDigitalRightsManagement(DRM)tosecureourvideostreams.
Overthepastyear,wevebeguntouseSecureHTTP(HTTPoverTLSorHTTPS)toencryptthe
transportofthevideocontentaswell.Thishelpsprotectmemberprivacy,particularlywhenthe
networkisinsecureensuringthatourmembersaresafefromeavesdroppingbyanyonewho
mightwanttorecordtheirviewinghabits.

NetflixOpenConnectservesover125millionhoursofcontentperday,allaroundtheworld.Given
ourscale,addingtheoverheadofTLSencryptioncalculationstoourvideostreamtransporthad
thepotentialtogreatlyreducetheefficiencyofourglobalinfrastructure.Wetakethisefficiency
seriously,sowehadtofindcreativewaystoenhancethesoftwareonourOCAstoaccomplish
thisobjective.

Wewilldescribeourworkinthesethreemainareas:

RSSFeed

AbouttheNetflixTechBlog
ThisisaNetflixblogfocusedon
technologyandtechnologyissues.
We'llshareourperspectives,
decisionsandchallengesregarding
thesoftwarewebuildanduseto
createtheNetflixservice.

BlogArchive
2016(38)
August(4)
ProtectingNetflix
ViewingPrivacyat
Scale

Determiningtheidealcipherforbulkencryption

IntroducingWinston
Eventdriven
Diagnosticand...

Findingthebestimplementationofthechosencipher

VizceralOpenSource

Exploringwaystoimprovethedatapathtoandfromthecipherimplementation

NetflixBilling
MigrationtoAWS
PartIII

CipherEvaluation

July(5)
June(4)
May(6)

WeevaluatedavailableandapplicableciphersanddecidedtoprimarilyusetheAdvanced
EncryptionStandard(AES)cipherinGalois/CounterMode(GCM),availablestartinginTLS1.2.
WechoseAESCGMovertheCipherBlockChaining(CBC)method,whichcomesatahigher
computationalcost.TheAESGCMcipheralgorithmencryptsandauthenticatesthemessage
simultaneouslyasopposedtoAESCBC,whichrequiresanadditionalpassoverthedatato
generatekeyedhashmessageauthenticationcode(HMAC).CBCcanstillbeusedasafallback
forclientsthatcannotsupportthepreferredmethod.
http://techblog.netflix.com/2016/08/protectingnetflixviewingprivacyat.html

April(6)
March(7)
February(4)
January(2)
2015(50)
2014(37)

1/9

8/9/2016

TheNetflixTechBlog:ProtectingNetflixViewingPrivacyatScale
2013(52)

AllrevisionsofOpenConnectAppliancesalsohaveIntelCPUsthatsupportAESNI,the
extensiontothex86instructionsetdesignedtoimproveencryptionanddecryptionperformance.
WeneededtodeterminethebestimplementationofAESGCMwiththeAESNIinstructionset,so
weinvestigatedalternativestoOpenSSL,includingBoringSSLandtheIntelIntelligentStorage
AccelerationLibrary(ISAL).

AdditionalOptimizations

2012(37)
2011(17)
2010(8)

Labels
cloudarchitecture(4)

NetflixandNGINXhadpreviouslyworkedtogethertoimproveourHTTPclientrequestand
responsetimeviatheuseofsendfilecallstoperformazerocopydataflowfromstorage(HDD
orSSD)tonetworksocket,keepingthedatainthekernelmemoryaddressspaceandrelieving
someoftheCPUburden.TheNetflixteamspecificallyaddedtheabilitytomakethesendfile
callsasynchronousfurtherreducingthedatapathandenablingmoresimultaneousconnections.

A/BTesting(3)
acceleratedcompositing(2)
adwords(1)
Aegisthus(1)
AESGCM(1)
AESNI(1)
algorithms(4)
aminator(2)
analytics(5)
Android(3)
angular(1)
animations(1)
ApacheMesos(1)
api(16)
appender(1)
AppsAndDevices(3)

However,TLSfunctionality,whichrequiresthedatatobepassedtotheapplicationlayer,was
incompatiblewiththesendfileapproach.

Archaius(2)
architecturaldesign(1)
architecture(2)
artwork(1)
Asgard(1)
Astyanax(4)
authentication(1)
automation(4)
autoscaling(3)
availability(4)
AWS(31)
bake(1)
benchmark(2)
bigdata(11)

ToretainthebenefitsofthesendfilemodelwhileaddingTLSfunctionality,wedesigneda
hybridTLSschemewherebysessionmanagementstaysintheapplicationspace,butthebulk
encryptionisinsertedintothesendfiledatapipelineinthekernel.Thisextendssendfileto
supportencryptingdataforTLS/SSLconnections.

billing(4)
Blitz4j(1)
build(4)
Cable(1)
caching(5)
Cassandra(14)
chaosengineering(1)
chaosmonkey(5)

http://techblog.netflix.com/2016/08/protectingnetflixviewingprivacyat.html

2/9

8/9/2016

TheNetflixTechBlog:ProtectingNetflixViewingPrivacyatScale
chukwa(1)
ci(1)
classloaders(1)
Clojure(1)
cloud(27)
cloudarchitecture(19)
cloudprize(3)
CO2(1)
collection(1)
complexeventprocessing(1)
computervision(2)

Wealsomadesomeimportantfixestoourearlierdatapathimplementation,includingeliminating
theneedtorepeatedlytraversembuflinkedliststogainaddressesforencryption.

concurrency(1)
configuration(2)

TestingandResults

configurationmanagement(2)
conformitymonkey(1)

WetestedtheBoringSSLandISALAESGCMimplementationswithoursendfile
improvementsagainstabaselineofOpenSSL(withnosendfilechanges),undertypicalNetflix
trafficconditionsonthreedifferentOCAhardwaretypes.OurchangesinboththeBoringSSLand
ISALtestsituationssignificantlyincreasedbothCPUutilizationandbandwidthoverbaseline
increasingperformancebyupto30%,dependingontheOCAhardwareversion.Wechosethe
ISALcipherimplementation,whichhadslightlybetterresults.Withtheseimprovementsinplace,
wecancontinuetheprocessofaddingTLStoourvideostreamsforclientsthatsupportit,without
sufferingprohibitiveperformancehits.

contentdelivery(1)
contentmetadata(2)
contentplatformengineering(3)
contentquality(1)
continuousdelivery(4)
coordination(2)
costmanagement(1)

Readmoredetailsinthispaperandthefollowuppaper.Wecontinuetoinvestigatenewandnovel
approachestomakingbothsecurityandperformanceareality.Ifthiskindofgroundbreakingwork
isupyouralley,checkoutourlatestjobopenings!

Cryptography(2)
CSS(2)
CUDA(1)

ByRandallStewart,AlexGutarin,andEllenLivengood
0Comments

crypto(1)

DaRE(1)

Sortby Oldest

dart(1)
data(1)

Addacomment...

datamigration(1)
datapipeline(5)
datascience(8)

FacebookCommentsPlugin

datavisualization(1)

Postedbytechwriterat8:00AM

database(7)

+4 Recommend this on Google

DataStax(2)

Labels:AESGCM,AESNI,encryption,FreeBSD,https,ISAL,NGINX,OpenConnect,optimization,
performance,security,sendfile,ssl,tls

deadlock(1)
deeplearning(1)
Denominator(2)
dependencyinjection(1)

Home

OlderPost

device(3)
deviceproliferation(1)
devops(3)
distributed(11)
DNS(1)
Docker(1)

http://techblog.netflix.com/2016/08/protectingnetflixviewingprivacyat.html

3/9

8/9/2016

TheNetflixTechBlog:ProtectingNetflixViewingPrivacyatScale
Dockerhub(1)
DSL(1)
Dyn(1)
DynECT(1)
Dynomite(1)
efficiency(2)
ElasticLoadBalancer(1)
elasticsearch(4)
ELB(1)
EMR(2)
encoding(6)
encryption(1)
energy(1)
eucalyptus(1)
eureka(2)
evcache(2)
Experimentation(2)
failover(2)
falcor(2)
faulttolerance(12)
flamegraphs(2)
Flow(1)
flux(1)
fmeasure(1)
footprint(1)
FreeBSD(1)
FRP(1)
functionalreactive(1)
garbage(1)
garbagecollection(1)
gc(1)
Genie(4)
git(1)
googlespreadsheet(1)
Governator(1)
GPU(2)
gradle(1)
green(1)
Groovy(1)
HackDay(3)
Hadoop(12)
HBase(1)
highvolume(4)
highvolumedistributedsystems
(11)

http://techblog.netflix.com/2016/08/protectingnetflixviewingprivacyat.html

4/9

8/9/2016

TheNetflixTechBlog:ProtectingNetflixViewingPrivacyatScale
Hive(2)
HTML5(8)
https(2)
Hystrix(5)
IBM(1)
ice(1)
images(1)
IMF(4)
IMSC(3)
infrastructure(2)
initialization(1)
innovation(3)
insights(1)
IntegrationTesting(1)
interprocesscommunication(1)
InteroperableMasterFormat(4)
iOS(1)
Ipv6(2)
ISAL(1)
isolation(1)
ISP(1)
java(5)
JavaScript(19)
jclouds(1)
jenkins(2)
kafka(4)
Karyon(2)
keystone(2)
lifecycle(1)
linux(2)
lipstick(2)
loadbalancing(3)
localization(1)
localizationplatformengineering(1)
locking(1)
locks(1)
log4j(1)
logging(2)
machinelearning(7)
Mantis(1)
MapReduce(1)
mediapipeline(2)
meetup(3)
memcache(2)
memcached(1)

http://techblog.netflix.com/2016/08/protectingnetflixviewingprivacyat.html

5/9

8/9/2016

TheNetflixTechBlog:ProtectingNetflixViewingPrivacyatScale
Meson(2)
messagesecuritylayer(1)
microservice(1)
migration(1)
Mobile(3)
modules(1)
monitoring(1)
msl(1)
nebula(1)
negativekeywords(1)
Netflix(19)
NetflixAPI(8)
netflixgraph(1)
NetflixOSS(15)
NetflixOSS(12)
neuralnetworks(1)
NGINX(1)
node.js(4)
NoSQL(5)
observability(1)
OpenConnect(1)
Opensource(11)
operationalexcellence(2)
operationalinsight(2)
operationalvisibility(1)
optimization(3)
Originals(2)
OSS(5)
outage(1)
pagegeneration(1)
payments(1)
Paypal(1)
performance(25)
personalization(7)
personalizationinfrastructure(1)
phone(1)
Photon(1)
Pig(4)
pipeline(1)
pki(1)
Playback(2)
precision(1)
prediction(2)
predictivemodeling(3)
Presto(2)

http://techblog.netflix.com/2016/08/protectingnetflixviewingprivacyat.html

6/9

8/9/2016

TheNetflixTechBlog:ProtectingNetflixViewingPrivacyatScale
prize(1)
prs(1)
pubsub(1)
pytheas(1)
python(3)
Quality(3)
qualitycontrol(3)
qualitymetric(2)
queries(1)
query(1)
rca(2)
React(3)
ReactiveProgramming(2)
realtimeinsights(2)
realtimestreaming(3)
recall(1)
Recipe(1)
recommendations(9)
Redis(4)
reinvent(2)
relevancy(1)
reliability(7)
remoteprocedurecalls(1)
renewable(1)
research(2)
resiliency(7)
REST(3)
Ribbon(2)
RiotGames(1)
rootcauseanalysis(2)
Route53(1)
ruleengine(1)
Rx(2)
Samza(1)
scalability(12)
scale(1)
scriptinglibrary(1)
search(4)
security(9)
sendfile(1)
Servo(1)
sharedlibraries(1)
simianarmy(5)
SimpleDB(3)
sitereliability(1)

http://techblog.netflix.com/2016/08/protectingnetflixviewingprivacyat.html

7/9

8/9/2016

TheNetflixTechBlog:ProtectingNetflixViewingPrivacyatScale
solr(1)
spark(2)
spinnaker(1)
sqoop(1)
ssd(2)
ssl(3)
STAASH(1)
Stamos(1)
streamprocessing(4)
streaming(2)
suro(1)
SWF(1)
synchronization(1)
tablet(2)
test(2)
testability(1)
testing(2)
TimedText(1)
Titus(1)
tls(3)
traffic(1)
trafficoptimization(1)
TTML(3)
TV(5)
UI(15)
UltraDNS(1)
unittest(3)
uptime(2)
userinterface(5)
Velocity(1)
videoquality(3)
visualization(1)
vizceral(1)
WebKit(3)
websockets(1)
WiiU(1)
windows(1)
winner(1)
winners(1)
Winston(1)
workflow(1)
workshop(1)
ZeroToDocker(1)
ZooKeeper(1)

http://techblog.netflix.com/2016/08/protectingnetflixviewingprivacyat.html

8/9

8/9/2016

TheNetflixTechBlog:ProtectingNetflixViewingPrivacyatScale
zuul(1)

TermsofUse|Privacy|CookiePreferences

AwesomeInc.template.PoweredbyBlogger.

http://techblog.netflix.com/2016/08/protectingnetflixviewingprivacyat.html

9/9