LuaSocket

behind the scenes

Diego Nehab
Short Bio

• Graduated from PUC in CS & E, 1999;

• Worked in Tecgraf 1995-2002;
• MSc in PL with Roberto, 2001;
• 3rd year PhD candidate at Princeton;
• Computer Graphics.
Outline of talk

• A few historical notes

• Case study: SMTP support
• Protocol abstraction
• Message abstraction
• Implementation highlights
• Conclusions
Historical notes

• 1.0, 1999, 1.5k C, 200 man

• 1.1, 2000, 1.5k C, 1.3k Lua, 500 man
• added protocol support for HTTP, SMTP, FTP
• 1.2, 2001, 2k C, 1.3k Lua, 900 man
• buffered input and non-blocking I/O
• UDP support
• object oriented syntax
Historical notes

• 1.3, 2001, 2.3k C, 1.6k Lua, 1.2k man

• streaming with callbacks
• added select function
• 1.4, 2001-2, 2.2k C, 2.2k Lua, 1.9k man
• LTN7
• added URL module
• named parameters
Current version

• 2.0, 2005, 4.6k C, 2.5k Lua, 4.7k man

• Extensible C architecture, split in modules
• LTN12 (sources, sinks and filters)
• MIME support (partial but honest) David Burgess

• Multipart messages support

• LTN13 (finalized exceptions)
• Package proposal
• Improved non-blocking code, robust to signals...
Outline of talk

• A few historical notes

• Case study: SMTP support
• Protocol abstraction
• Message abstraction
• Implementation highlights
• Conclusions
SMTP (RFC2821)
[lua:roberto] telnet mail.tecgraf.puc-rio.br 25
220 tecgraf.puc-rio.br ESMTP Sendmail 8.9.3/8.9.3
helo lua
250 tecgraf.puc-rio.br Hello lua, pleased to meet you
mail from: <roberto@inf.puc-rio.br> from
250 <roberto@inf.puc-rio.br>... Sender ok
rcpt to: <diego@tecgraf.puc-rio.br> rcpt
250 <diego@tecgraf.puc-rio.br>... Recipient ok
data
354 Enter mail, end with "." on a line by itself
Subject: World domination: instructions.
body
Commence stage two.
.
250 RAA10452 Message accepted for delivery
quit
221 tecgraf.puc-rio.br closing connection
Protocol abstraction
status, error = smtp.send {
from = "<roberto@inf.puc-rio.br>",
rcpt = "<diego@tecgraf.puc-rio.br>",
body = "Subject: World domination: instructions.\r\n\r\n" ..
"Comence stage two."
}

• What if body is large?

LTN12 sources

• Use callback function that produces data;

• Returns one chunk each time called;
• Signals termination returning nil.
function ltn12.source.file(handle)
return function()
local chunk = handle:read(BLOCKSIZE)
if not chunk then handle:close() end
return chunk
end
end
Using sources
status, message = smtp.send {
from = "<roberto@inf.puc-rio.br>",
rcpt = "<diego@tecgraf.puc-rio.br>",
body = ltn12.source.file(io.open("/mail/body", "r"))
}

• What if body is complicated?

Message Format (RFC2822)
From: Roberto Ierusalimschy <roberto@inf.puc-rio.br>
To: Diego Nehab <diego@tecgraf.puc-rio.br>
headers
Subject: World domination: roadmap.
Content-Type: multipart/mixed; boundary=part

This message contains attachments

--part
Content-Type: text/plain headers
part 1
Please see attached roadmap. body
body
--part
Content-Type: text/html; name="roadmap.html"
... part 2
--part--
Message abstraction
declaration = {
headers = {
subject = "World domination",
from = "Roberto <roberto@inf.puc-rio.br>",
to = "Diego <diego@tecgraf.puc-rio.br>"
},
preamble = "This message contains attachments.",
[1] = {
headers = { ... },
body = "Please see attatched roadmap."
},
[2] = {
headers = { ... },
body = ltn12.source.file(io.open("/plans/roadmap.html", "r"))
}
}
Our message API
status, message = smtp.send {
from = "<roberto@inf.puc-rio.br>",
rcpt = "<diego@tecgraf.puc-rio.br>",
body = smtp.message(declaration)
}

• Transform declaration into an LTN12 source;

• Pass source as body to sending function.
How hard is it?

• Message structure is recursive;

• Need to return chunks but mantain context;
• Nightmare to write in C!;
• Use coroutines;
• Write function recursively, naturally;
• Call yield with each chunk;
• Next call resumes wherever we left.
Zoom in on attachments
[2] = {
headers = {
["content-type"] = 'text/html; name="roadmap.html"',
["content-disposition"] = 'attachment; filename ="roadmap.html"'
},
body = ltn12.source.file(io.open("/plans/roadmap.html", "r"))
}

• Would like to send PDF;

• Binary data has to be encoded (Base64);
• Want to encode on-the-fly.
LTN12 filters and chains

• Filters process data one chunk at a time;

• MIME module provides common filters:
• base64, quoted-printable, stuffing, line-wrap...
• Can chain two filters together: factory
• Produce a filter with the composite effect
• Can chain a filter with a source: factory
• Produce a source that returns filtered data.
Zoom in on attachments
[2] = {
headers = {
["content-type"] = 'application/pdf; name="roadmap.pdf"',
["content-disposition"] = 'attachment; filename ="roadmap.pdf"',
["content-description"] = 'Detailed world domination plan',
["content-transfer-encoding"] = 'BASE64'
},
body = ltn12.source.chain(
ltn12.source.file(io.open("/plans/roadmap.pdf", "r")),
ltn12.filter.chain(
mime.encode("base64"),
mime.wrap("base64")
)
)
}
Creating filters: high-level

• Chunks can be broken arbitrarily;

• Filters have to keep context between calls;
function ltn12.filter.cycle(low, ctx, extra)
return function(chunk)
local ret
ret, ctx = low(ctx, chunk, extra)
return ret
end
end
function mime.normalize(marker)
return ltn12.filter.cycle(mime.eol, 0, marker)
end
Creating filters: low-level
int eol(lua_State *L) {
int ctx = luaL_checkint(L, 1);
size_t isize = 0;
const char *input = luaL_optlstring(L, 2, NULL, &isize);
const char *last = input + isize;
const char *marker = luaL_optstring(L, 3, CRLF);
luaL_Buffer buffer;
luaL_buffinit(L, &buffer);
while (input < last)
ctx = translate(*input++, ctx, marker, &buffer);
luaL_pushresult(&buffer);
lua_pushnumber(L, ctx);
return 2;
}
Creating filters: low-level
#define candidate(c) (c == CR || c == LF)
int translate(int c, int last, const char *mark, luaL_Buffer *buffer) {
if (candidate(c)) {
if (candidate(last)) {
if (c == last) luaL_addstring(buffer, mark);
return 0;
} else {
luaL_addstring(buffer, mark);
return c;
}
} else {
luaL_putchar(buffer, c);
return 0;
}
}
SMTP dependencies

socket tp

smtp ltn12

mime
Error checking

• Function return convention

• Return nil, followed by message on error;
function metat.__index:greet(domain)
local r, e = self.tp:check("2..")
if not r then return nil, e end
r, e = self.tp:command("HELO", domain)
if not r then return nil, e end
return self.tp:check("2..")
end

• Tedious, error prone, virotic, not finalized.

LTN13 exceptions

• try = newtry(finalizer): factory;

• On success, try returns all arguments;
• On failure, throws the second argument;
• Calls finalizer before raising the exception.
• foo = protect(bar): factory;
• foo executes bar in a protected environment;
• Returns nil followed by any thrown error.
No 'if' statements
function metat.__index:greet(domain)
self.try(self.tp:check("2.."))
self.try(self.tp:command("HELO", domain))
return self.try(self.tp:check("2.."))
end

• Internal functions throw exceptions;

• try calls tp.close() on error;
• External functions can be protected.
Conclusions

• Hope you like our API, we do;

• It is easy to implement;
• Function factories + closures, coroutines
• It is fast;
• Time critical in C, management in Lua;
• Questions?