• Worked in Tecgraf 1995-2002; • MSc in PL with Roberto, 2001; • 3rd year PhD candidate at Princeton; • Computer Graphics. Outline of talk
• A few historical notes
• Case study: SMTP support • Protocol abstraction • Message abstraction • Implementation highlights • Conclusions Historical notes
• 1.0, 1999, 1.5k C, 200 man
• 1.1, 2000, 1.5k C, 1.3k Lua, 500 man • added protocol support for HTTP, SMTP, FTP • 1.2, 2001, 2k C, 1.3k Lua, 900 man • buffered input and non-blocking I/O • UDP support • object oriented syntax Historical notes
• 1.3, 2001, 2.3k C, 1.6k Lua, 1.2k man
• streaming with callbacks • added select function • 1.4, 2001-2, 2.2k C, 2.2k Lua, 1.9k man • LTN7 • added URL module • named parameters Current version
• 2.0, 2005, 4.6k C, 2.5k Lua, 4.7k man
• Extensible C architecture, split in modules • LTN12 (sources, sinks and filters) • MIME support (partial but honest) David Burgess
• Multipart messages support
• LTN13 (finalized exceptions) • Package proposal • Improved non-blocking code, robust to signals... Outline of talk
• A few historical notes
• Case study: SMTP support • Protocol abstraction • Message abstraction • Implementation highlights • Conclusions SMTP (RFC2821) [lua:roberto] telnet mail.tecgraf.puc-rio.br 25 220 tecgraf.puc-rio.br ESMTP Sendmail 8.9.3/8.9.3 helo lua 250 tecgraf.puc-rio.br Hello lua, pleased to meet you mail from: <roberto@inf.puc-rio.br> from 250 <roberto@inf.puc-rio.br>... Sender ok rcpt to: <diego@tecgraf.puc-rio.br> rcpt 250 <diego@tecgraf.puc-rio.br>... Recipient ok data 354 Enter mail, end with "." on a line by itself Subject: World domination: instructions. body Commence stage two. . 250 RAA10452 Message accepted for delivery quit 221 tecgraf.puc-rio.br closing connection Protocol abstraction status, error = smtp.send { from = "<roberto@inf.puc-rio.br>", rcpt = "<diego@tecgraf.puc-rio.br>", body = "Subject: World domination: instructions.\r\n\r\n" .. "Comence stage two." }
• What if body is large?
LTN12 sources
• Use callback function that produces data;
• Returns one chunk each time called; • Signals termination returning nil. function ltn12.source.file(handle) return function() local chunk = handle:read(BLOCKSIZE) if not chunk then handle:close() end return chunk end end Using sources status, message = smtp.send { from = "<roberto@inf.puc-rio.br>", rcpt = "<diego@tecgraf.puc-rio.br>", body = ltn12.source.file(io.open("/mail/body", "r")) }
• What if body is complicated?
Message Format (RFC2822) From: Roberto Ierusalimschy <roberto@inf.puc-rio.br> To: Diego Nehab <diego@tecgraf.puc-rio.br> headers Subject: World domination: roadmap. Content-Type: multipart/mixed; boundary=part
This message contains attachments
--part Content-Type: text/plain headers part 1 Please see attached roadmap. body body --part Content-Type: text/html; name="roadmap.html" ... part 2 --part-- Message abstraction declaration = { headers = { subject = "World domination", from = "Roberto <roberto@inf.puc-rio.br>", to = "Diego <diego@tecgraf.puc-rio.br>" }, preamble = "This message contains attachments.", [1] = { headers = { ... }, body = "Please see attatched roadmap." }, [2] = { headers = { ... }, body = ltn12.source.file(io.open("/plans/roadmap.html", "r")) } } Our message API status, message = smtp.send { from = "<roberto@inf.puc-rio.br>", rcpt = "<diego@tecgraf.puc-rio.br>", body = smtp.message(declaration) }
• Transform declaration into an LTN12 source;
• Pass source as body to sending function. How hard is it?
• Message structure is recursive;
• Need to return chunks but mantain context; • Nightmare to write in C!; • Use coroutines; • Write function recursively, naturally; • Call yield with each chunk; • Next call resumes wherever we left. Zoom in on attachments [2] = { headers = { ["content-type"] = 'text/html; name="roadmap.html"', ["content-disposition"] = 'attachment; filename ="roadmap.html"' }, body = ltn12.source.file(io.open("/plans/roadmap.html", "r")) }
• Would like to send PDF;
• Binary data has to be encoded (Base64); • Want to encode on-the-fly. LTN12 filters and chains
• Filters process data one chunk at a time;
• MIME module provides common filters: • base64, quoted-printable, stuffing, line-wrap... • Can chain two filters together: factory • Produce a filter with the composite effect • Can chain a filter with a source: factory • Produce a source that returns filtered data. Zoom in on attachments [2] = { headers = { ["content-type"] = 'application/pdf; name="roadmap.pdf"', ["content-disposition"] = 'attachment; filename ="roadmap.pdf"', ["content-description"] = 'Detailed world domination plan', ["content-transfer-encoding"] = 'BASE64' }, body = ltn12.source.chain( ltn12.source.file(io.open("/plans/roadmap.pdf", "r")), ltn12.filter.chain( mime.encode("base64"), mime.wrap("base64") ) ) } Creating filters: high-level
• Chunks can be broken arbitrarily;
• Filters have to keep context between calls; function ltn12.filter.cycle(low, ctx, extra) return function(chunk) local ret ret, ctx = low(ctx, chunk, extra) return ret end end function mime.normalize(marker) return ltn12.filter.cycle(mime.eol, 0, marker) end Creating filters: low-level int eol(lua_State *L) { int ctx = luaL_checkint(L, 1); size_t isize = 0; const char *input = luaL_optlstring(L, 2, NULL, &isize); const char *last = input + isize; const char *marker = luaL_optstring(L, 3, CRLF); luaL_Buffer buffer; luaL_buffinit(L, &buffer); while (input < last) ctx = translate(*input++, ctx, marker, &buffer); luaL_pushresult(&buffer); lua_pushnumber(L, ctx); return 2; } Creating filters: low-level #define candidate(c) (c == CR || c == LF) int translate(int c, int last, const char *mark, luaL_Buffer *buffer) { if (candidate(c)) { if (candidate(last)) { if (c == last) luaL_addstring(buffer, mark); return 0; } else { luaL_addstring(buffer, mark); return c; } } else { luaL_putchar(buffer, c); return 0; } } SMTP dependencies
socket tp
smtp ltn12
mime Error checking
• Function return convention
• Return nil, followed by message on error; function metat.__index:greet(domain) local r, e = self.tp:check("2..") if not r then return nil, e end r, e = self.tp:command("HELO", domain) if not r then return nil, e end return self.tp:check("2..") end
• Tedious, error prone, virotic, not finalized.
LTN13 exceptions
• try = newtry(finalizer): factory;
• On success, try returns all arguments; • On failure, throws the second argument; • Calls finalizer before raising the exception. • foo = protect(bar): factory; • foo executes bar in a protected environment; • Returns nil followed by any thrown error. No 'if' statements function metat.__index:greet(domain) self.try(self.tp:check("2..")) self.try(self.tp:command("HELO", domain)) return self.try(self.tp:check("2..")) end
• Internal functions throw exceptions;
• try calls tp.close() on error; • External functions can be protected. Conclusions
• Hope you like our API, we do;
• It is easy to implement; • Function factories + closures, coroutines • It is fast; • Time critical in C, management in Lua; • Questions?