You are on page 1of 26

HS2LIB TUTORIAL

AUTOMATING THE CREATION OF STATIC LIBRARIES FROM HASKELL PROGRAMS

Tamar Christina

tamar@zhox.com

http://hackage.haskell.org/package/Hs2lib

September 4, 2011

PART OF THE VISUAL HASKELL 2010 TOOLCHAIN

This is a shortened version of the manual showing only the introduction and examples. Full manual to follow.

Copyright © 2011 Tamar Christina

http://code.zhox.com/haskell/hs2lib

Licensed under the BSD License, Version 3.0 (the “License”); you may not use this le except in compliance with the License. You may obtain a copy of the License at http://www.opensource.org/licenses/bsd-license. php. Unless required by applicable law or agreed to in writing, software distributed under the License is dis- tributed on an “as is” basis, without warranties or conditions of any kind, either express or implied. See the License for the speci c language governing permissions and limitations under the License.

First printing, September 2011

Contents

1 Introduction

 

7

1.1

Motivation

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

7

2 Examples

 

9

2.1 Simple Arithmetic

 

.

.

.

.

.

.

.

.

.

.

.

.

.

9

2.2 User data types

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

23

2.3 Callbacks

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

24

Sessions

2.4 .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

25

CONTENTS 5

Release Notes

Version 0.5.5

- (Started) implementing a better test mechanism

Version 0.5.4

- fixed a major marshalling bug having to do with lists of pointer types.

Version 0.5.3

- Fixed an error with parsing pragmas

- Renamed pragma hs2c# to hs2cs

- Fixed an alignment issue with stdcall

- Fixed an issue with lists and type synonyms

Version 0.5.2

- No longer frees Strings, in order to prevent heap corruptions on calls from C#.

Version 0.5.1

- Library bug fix which cauzed an error in marshalling

Version 0.5.0

- Added support for memory leak tracing with the --debug flag

% Added support for qualified importing. It is now possible to export values and types with the same name. (unfinished)

% - Added the ability to export polymorphic types. (unfinished)

% - Added support for infix constructors (unfinished)

- Generates free functions StablePtr

Version 0.4.8

- Major restructuring of code to allow users to extend/override the default type conversions of the tool.

- Support for custom type translations

Version 0.4.5

- Brand new Haskell code generators

- Support for lists inside an IO wrapper (e.g. IO [Int])

- Brand new implementation in NativeMapping

- Avoids unsafePerformIO as much as possible

Version 0.4.4

- Fixed Include paths

- NO support for datastructures with multiple constructors where one constructor has the same name as the datatype (codegen naming limitation)

- Fixed codegen issue with callbacks

- Added better IO Error handling

Version 0.4.3 (Initial Release)

6

hs2lib manual

- NO support for infix constructors

- NO support for lists inside Applied types (e.g. Maybe [String])

- NO support for lists inside tuples (e.g. (Int, [String]))

- Does not allow for custom translation of types.

- NO support for quantified types. (e.g. M.Map)

1

Introduction

Creating a static library from a Haskell program is no easy task. It re- quires you to create marshalling code 1 to translate all your Haskell structures. Create foreign export declarations, write some initialization code and more. Hs2Lib aims to make this much easier by providing very simple annotations that can be used to mark functions that needs to be exported. Calling C functions from Haskell is very popular and has a lot of tool- ing support. The tool c2hs generates the appropriate Haskell data types and bindings from a C header le. Hs2Lib however does the inverse and much more. Since this tool is part of the back end of Visual Haskell 2010, it is essential that we be able to generate C# code as well. The tool itself is platform independent, but has mostly been tested only on windows.

1.1 Motivation

1 De ning instances of the Storable instance for your datatypes. More on this in Chapter ??

Throughout the years it has become increasingly easier to create shared libraries from Haskell programs when using GHC 2 . The task however 2 remains quite involved. The canonical example is the Adder program 3 : 3 Syntax highlighting for Haskell

module Adder where

adder ::Int Int IO Int adder x y return x y

foreign export stdcall adder ::Int Int IO Int

provided by Alessandro Ver- meulen’s lhs2tex-hl. Check it out at

http://alessandrovermeulen.me/projects/lhs2texhl/

Compiling the .DLL Adder.dll requires just three short commands:

ghc -c Adder.hs

ghc -c

ghc -shared -o Adder.dll Adder.o Adder_stub.o StartEnd.o

StartEnd.c 4

4 StartEnd.c contains RTS Initialization and termination code. More about this in Chapter ??

8

hs2lib manual

This is somewhat disingenuous, the Haskell FFI report 5 speci es Char ,Int,Double,Float and Bool as basic foreign types exported by the prelude. The Adder example is thus limited to using types that are

prede ned to be a Foreign type. No custom marshallers are de ned 6 since 6 marshalling (similar to serialization) is

we do not export any user types. The steps that in general need to be taken to compile an arbitrary Haskell le are in a nutshell:

5

the process of transforming the memory representation of an object to a data for- mat suitable for storage or transmission. It is typically used when data must be moved between di erent parts of a com- puter program or from one program to another.

RTS FFI Storable GHC DLL DEF LIB
RTS
FFI
Storable
GHC
DLL
DEF
LIB

Figure 1.1: Compilation steps to produce a static lib

RTS De nes the runtime system initialization and termination function. Without this le we would have no way of starting and stopping the Haskell runtime.

FFI Declare the appropriate foreign export de nitions. Also create wrappers over the to be exported functions, which then exposes them using types that are supported by the Foreign Function Interface.

Storable De ne marshalling code for user data types. This is done by providing a proper Storable instance for every data type that needs to be exported.

DEF (optional) By default, everything that is a function is exported by GHC in the Exports table by GHC. This includes closures etc. To hide all the noise we can de ne our own DEF le, which then states which functions are to be put in the static lib’s export table.

LIB (optional) If we want to use MSVC++ and compile C++ code which uses the Haskell DLL, then we also need a .LIB le for the linker, which is created in this step

GHC The nal step is to use a set of GHC commands to link and compile everything together into a neat little package.

Doing these steps takes a long time. They are also quite repetitive and easy to automate. Which is exactly what Hs2Lib does. 7

7 NOTE: This tool does not use, nor keep the stubs generated from GHC. It produces its own includes.

2

Examples

This chapter shows 4 examples of small case studies to illustrate how to use Hs2lib. The rst one will be rather elaborate, showing how to call this tool from all three code generators. The rest will all be a bit more compact.

2.1 Simple Arithmetic

The rst example involves a set of simple arithmetic functions.

module Arith where

-- @@ Export summerize:: Int Int Int summerize x y sum x y

-- @@ Export

single::Int Int

single x 1

x

Figure 2.1: Simple Haskell Arithmetic

The code is quite simple. summerize takes two list of integers and calculates their collective sum, whereas single takes an integer x and returns a list which contains the enumeration from 1 to x. The functions to be annotated have already been marked using the export annotation. In order to compile this example the only command needed is hs2lib Arith.

PS C:\Examples> hs2lib Arith

10

hs2lib manual

Linking main.exe

The directory should now contain the following les

Directory: C:\Examples

Mode

LastWriteTime

Length Name

----

-------------

------ ----

-a---

5/30/2011

1:19 AM

534 Arith.hi

-a---

5/30/2011

1:17 AM

159 Arith.hs

-a---

5/30/2011

1:19 AM

1973 Arith.o

-a---

5/30/2011

1:20 AM

6874624 Hs2lib.dll

-a---

5/30/2011

1:20 AM

711 Hs2lib.h

-a---

5/30/2011

1:20 AM

1250 Hs2lib_FFI.h

-a---

5/30/2011

1:20 AM

5630 HSHs2lib.a

The default namespace used by the tool is "Hs2lib" 1 . The library pro- 1 This can be changed by using the -n ag

duced has also already been stripped 2 . By default, manual RTS initialization needs to be done. If we were to look at the exports table for Hs2lib.dll 3 we would see the following output

Microsoft (R) COFF/PE Dumper Version 10.00.40219.01

Copyright (C) Microsoft Corporation.

All rights reserved.

Dump of file Hs2lib.dll

File Type: DLL

2 The only extra shrinkage that can be gained now is by packing the le using something like upx. The smaller size comes with some slight overhead in start-up time. UPX will not be executed by Hs2lib 3 dumpbin /EXPORTS Hs2lib.dll

Section contains the following exports for Hs2lib.dll

00000000 characteristics 4DE2D4AC time date stamp Mon May 30 01:20:12 2011 0.00 version

1

ordinal base

8

number of functions

8

number of names

ordinal hint RVA

name

1 0 00002299 HsEnd@0

2 1 00002258 HsStart@0

3 2 00002299 _HsEnd@0

4 3 00002258 _HsStart@0

5 4 0000204C _single@8

6 5 00001F54 _summerize@16

7 6 0000204C single@8

8 7 00001F54 summerize@16

examples 11

We can see that thanks to the .DEF 4 le only the functions we wanted 4 See Section ?? for details have been exported. Along with the RTS initialization functions HsStart and HsEnd.

C With GCC and Netbeans

The rst call we’ll make is from C, using the gcc compiler and the Net- beans 6.8 IDE. We start o by creating an new C project as depicted in gure 2.2

by creating an new C project as depicted in gure 2.2 Figure 2.2: New project dialog

Figure 2.2: New project dialog of Netbeans 6.8

In the le main.c add the following code:

#include <stdio.h> #include <stdlib.h> #include "Hs2lib_FFI.h"

int main(int argc, char** argv) {

HsStart();

int* foo = (int*)malloc(sizeof(int)*3); =

foo[1]

foo[0]

2;

5;

=

foo[2] = 10;

printf("Sum result: %d\n", summerize(3, foo, 3, foo));

free(foo);

int count; int* bar = single(8, &count);

12

hs2lib manual

printf("Single result: (%d)\n", count);

int i;

for(i

=

0;

i

< count; i++)

printf("\t%d\n", bar[i]);

free(bar);

HsEnd();

return (EXIT_SUCCESS);

}

This test tests the functions summerize and single. It also tests the ability to pass lists back and forth. If we try to run this project building will fail. This is because there’s still some settings we need to change. So let’s open up the project settings dialogue.

to change. So let’s open up the project settings dialogue. Figure 2.3: Linker dialog of Netbeans

Figure 2.3: Linker dialog of Netbeans 6.8

The rst thing we need to change is the linker settings. In particu- lar the Additional Library Directories and Libraries values need to be changed to the appropriate values. 5 In addition to the Linker, the Include Directories of the C Compiler settings also have to be changed.

5 In this case, C:\Examples and HSHs2lib respectively.

examples 13

examples 13 Figure 2.4: C Compiler Include Path dialog of Netbeans 6.8 As mentioned before, Hs2lib

Figure 2.4: C Compiler Include Path dialog of Netbeans 6.8

As mentioned before, Hs2lib comes with a set a prede ned includes that support the generated code. These les are located in

%AppData%\cabal\Hs2lib-version \Includes in windows 6 . Next 6 Note that most compilers don’t expand

to the Hs2lib include directory, we also need to include the folder in which the generated header les were placed for the current project 7 . The last thing that we need to change is the Run Directory. This sets the CWD of the executable. This needs to be set to the same folder which has our generated .DLL le, so that it is now in the path of the executable.

this path. So It’s recommended to expand them before hand and then add them to the Include Directories. You can do this by just opening up the folder in explorer, which will show you the full path. 7 C:\Examples in this case

will show you the full path. 7 C:\Examples in this case Figure 2.5: Run CWD Path

Figure 2.5: Run CWD Path dialog of Netbeans 6.8

The project can now be compiled and run, if all goes well, the follow- ing output is displayed:

14

hs2lib manual

14 hs2lib manual Figure 2.6: Arith results of Netbeans 6.8 Microsoft Visual C++ This section shows

Figure 2.6: Arith results of Netbeans 6.8

Microsoft Visual C++

This section shows how to use the generated dll from a C++ compiler such as Microsoft’s Visual C++. While the C compiler used the generated ".a" le, the MSVC++ compiler requires a di erent interface le: A Library import le (.lib). This can be easily generated, so lets start with that. If the tool lib.exe is on your path, Hs2lib can be instructed to gener- ate the .lib le automatically by using the ––lib or -M ags.

PS C:\Examples> hs2lib --lib Arith Linking main.exe Done.

There will now be a le Hs2lib.lib among the outputs. If lib.exe is not on your path then you need to manually generate the .lib le. To do this we need the .DEF le that was used by the tool. By default Hs2lib automatically deletes this le when performing its cleanup actions. To instruct it to keep this le, the ag ––incl-def or -d can be used.

PS C:\Examples> hs2lib --incl-def Arith Linking main.exe Done.

The next step is to open a Visual Studio command prompt and navigate to the folder containing the .DEF le. The .lib le can be easily generated then using the command lib /DEF:Hs2lib.def

C:\Examples>lib /DEF:Hs2lib.def

Microsoft (R) Library Manager Version 10.00.40219.01

Copyright (C) Microsoft Corporation.

All rights reserved.

LINK : warning LNK4068: /MACHINE not specified; defaulting to X86 Creating library Hs2lib.lib and object Hs2lib.exp

With the .lib le at hand, we can now create an empty MSVC++ Project.

examples 15

examples 15 Figure 2.7: New C++ Project Dialog Visual Studio 2010 We want a Win32 Console

Figure 2.7: New C++ Project Dialog Visual Studio 2010

We want a Win32 Console project, and also want an empty project.

a Win32 Console project, and also want an empty project. Figure 2.8: Empty C++ Project Wizard

Figure 2.8: Empty C++ Project Wizard Visual Studio 2010

The C++ Source le contains the following code:

#include <stdio.h> #include <stdlib.h> #include "Hs2lib_FFI.h"

int main() {

16

hs2lib manual

HsStart();

int* foo = new int[3]; =

foo[0]

foo[1]

foo[2] = 10;

2;

5;

=

printf("Sum result: %d\n", summerize(3, foo, 3, foo));

delete foo;

int count; int* bar = single(8, &count);

printf("Single result: (%d)\n", count);

int i;

for(i

=

0;

i

< count; i++)

printf("\t%d\n", bar[i]);

free(bar);

HsEnd();

return 0;

}

Just like before we test the exact same functions and expect the exact same results to be returned. Before we can run this however we need to change a few settings again. First of which, is the Linker dependencies.

a few settings again. First of which, is the Linker dependencies. Figure 2.9: Linked Input settings

Figure 2.9: Linked Input settings Visual Studio 2010

examples 17

We add Hs2lib.lib to the Additional Dependencies value of the Linker. This allows it to nd our exported functions. The next step is modify the Include- and Library Directories, we add the same values we added in Figure 2.4 and Figure 2.3.

the same values we added in Figure 2.4 and Figure 2.3 . Figure 2.10: VC++ Directories

Figure 2.10: VC++ Directories Visual Studio 2010

And lastly we need to modify the working directory using the same values as in Figure 2.5.

working directory using the same values as in Figure 2.5 . Figure 2.11: VC++ Current Working

Figure 2.11: VC++ Current Working Directory Visual Studio 2010

If we were to compile and run this project, We would unfortunately receive the following error message:

18

hs2lib manual

18 hs2lib manual Figure 2.12: VC++ Heap corruption error This is because of a mismatch between

Figure 2.12: VC++ Heap corruption error

This is because of a mismatch between memory allocators. The call

to free is corrupting the heap because the free function being used has

a di erent allocation style then the allocato(malloc) that reserved the

memory. The di erence is in the version of msvcrt.dll being used. GHC which uses GCC to link, links against the msvcrt.dll 8 which is the stan-

dard C runtime on windows. Unfortunately, every version of Microsoft Visual C++ brings with it a new version of the runtime. The version of visual studio being used here 9 uses the runtime msvcrt100.dll 10 . These 9 Visual Studio 2010 is the 10th version of

di erent versions of the runtime use di erent incompatible allocators.

The standard solution/convention is to always allocate and deallocate memory on the caller’s side 11 . I prefer not to use this approach, It would

require much more work to allocate memory for the Haskell data types. Instead, I provide in Hs2lib speci cally for this case a DLL 12 which provides us with the same free function being used by msvcrt.dll. It

is infact a simple redirect which just re-exports the free function. This

Support le exports a function freeNative which should be used to free

memory allocated in Haskell

visual studio.

11 It is probably for this reason that Win32 APIs require you to give a preallocated structure which will be lled in by the call instead of just returning a structure that it allocated.

12 WinDllSupport.dll

13 Include the le WinDllSupport.h and link to WinDllSupport.lib. Keep in mind that de calling convention for this DLL is Cdecl.

Land. 13

Using this alternative free function, our project nally runs correctly.

Land. 1 3 Using this alternative free function, our project nally runs correctly. Figure 2.13: VC++

Figure 2.13: VC++ Arith result

examples 19

Visual C#

The last part of this case study is using the C# code generator. Despise what intuition would suggest, this is actually quite straight forward. Hs2lib can generate the C# binding automatically for us. To do this, we need to pass it the ––c# ag.

PS C:\Examples> hs2lib --c# Arith Linking main.exe Done.

This generates a le Hs2lib.cs containing everything we need. We can now create the new empty C# project.

we need. We can now create the new empty C# project. Figure 2.14: C# New project

Figure 2.14: C# New project dialog

After we have our empty project, we can now add a reference to the FFI.dll managed dll. This contains a plethora of helper func-

tions to use the DLL from C# 14 . Everything both generated and in this 14 Highlights include a free function,

helper library will be in the WinDll namespace. The code that has just been generated can be found in WinDll.Generated and the types in WinDll.Generated.Types.

methods to read Maybe values and much more.

20

hs2lib manual

20 hs2lib manual Figure 2.15: C# Add Reference Dialog 1 5 After adding the references we

Figure 2.15: C# Add Reference Dialog

15

After adding the references we can add the two les we need for this project.

15 Remember to rst expand the path depicted by %AppData%.

Remember to rst expand the path depicted by %AppData%. Figure 2.16: C# Existing les Dialog We

Figure 2.16: C# Existing les Dialog

We need to add both the DLL and the Hs2lib.cs le that was gener- ated. By adding the DLL we can instruct MSBuild to copy the DLL to the output folder. This enables us to not have to set the Working directory of the project. First click on Hs2lib.DLL in the solution explorer and set its build action to Copy if newer.

examples 21

examples 21 Figure 2.17: C# Hs2lib.DLL property Dialog In Program.cs add the following lines using System

Figure 2.17: C# Hs2lib.DLL property Dialog

In Program.cs add the following lines

using System; using WinDLL.Generated; using WinDLL.Utils; using System.Runtime.InteropServices;

namespace CsArith

{

class Program

{

static void Main(string[] args)

{

Hs2lib.HsStart();

unsafe

{

IntPtr fooPtr = Marshal.AllocHGlobal(sizeof(int) * 3); int* foo = (int*)fooPtr.ToPointer();

foo[0] =

2;

foo[1] =

5;

foo[2] = 10;

Console.WriteLine("Sum result: {0}", Hs2lib.summerize(3, foo, 3, foo));

Marshal.FreeHGlobal(fooPtr);

int count; int* bar = Hs2lib.single(8, &count);

Console.WriteLine("Single result: ({0})", count);

22

hs2lib manual

int i;

for

(i

=

0;

i

< count; i++)

Console.WriteLine("\t{0}", bar[i]);

}

FFI.free(bar);

}

Hs2lib.HsEnd();

}

return;

}

Before we can compile this code we need to enable unsafe code compi- lation in the compiler. We can do this by going into the project’s property window.

can do this by going into the project’s property window. Figure 2.18: C# Project property Dialog

Figure 2.18: C# Project property Dialog

After this we can nally run the program, and the result should be the same as before.

examples 23

examples 23 Figure 2.19: C# Result Dialog 2.2 User data types This case studies deals with

Figure 2.19: C# Result Dialog

2.2 User data types

This case studies deals with how to export and read user de ned data types using Hs2lib. As with all preceding examples, we’ll only cover the C examples. The example used here is the evaluation of simple expres- sions.

module Expr where

data Expr Add Expr Expr Sub Expr Expr Mul Expr Expr Div Expr Expr Parens Expr Var Var

Val Int

Let String Expr Expr

type Var String

type Env Var,Int

-- @@ Export = eval

-- | Evaluate an expression, throwing an exception if the variable is not found. foldExpr ::Expr Int foldExpr foldE where foldE ::Env Expr Int

foldE env Add e 1 e 2 foldE env foldE env Sub e 1 e 2 foldE env foldE env Mul e 1 e 2 foldE env

foldE env Div e 1 e 2 foldE env e 1 div foldE env e 2 foldE env Parens e foldE env e foldE env Var nm case lookup nm env of Nothing error "Variable ’" nm "’ could not be found." Just val val foldE env Val val val

e 1 e 1

foldE env foldE env

e 2 e 2

e 1 foldE env e 2

24

hs2lib manual

foldE env Let x v e

let v

foldE env v

env x,v : env

in foldE env e

This time while compiling we’re going to instruct Hs2lib to rename the project. This can be done by setting the default namespace 16 16 This can be set with -n and will instead

of using the name Hs2lib for output les, use the name supplied for all les.

PS C:\Examples> hs2lib .\Expr.hs -n Eval Linking main.exe Done.

This creates a le Eval.DLL among other things. The exports of this DLL includes a function named eval. By specifying a name after the Export statement we can on an individual basis rename functions. I’ll just assume that the user is familiar with C and no explain how to call the function. By looking at the prototype of the function its clear that it’s pretty straight forward.

// eval :: Expr -> Int extern CALLTYPE(int) eval (Expr_t* arg1);

struct Expr { enum ListExpr tag; union ExprUnion* elt; Expr_t ;

}

2.3 Callbacks

Hs2lib supports Higher-order functions by making them callbacks.

-- @ Export myadd:: Int Int Int Int myadd $

Figure 2.20: Example exportable Higher-order function

The generated header le would contain the following de nition

// myadd :: CBF1 -> Int -> Int extern CALLTYPE(int) myadd (CBF1_t arg1, int arg2);

// Callback functions typedefs // type CBF1 = Int -> Int

typedef

stdcall

int (*CBF1_t)(int);

This is again rather straight forward, the higher-ordered arguments just become function pointers. One important thing is, if you use any lists or IO action in the higher-ordered argument. (e.g. Int Int Int Int) then the argument must be in IO (e.g. Int IO Int Int IO Int).

examples 25

The reason is that creating and reading arrays to lists is an IO oper- ation. If you omit this the tool will still compile, but be forced to use unsafePerformIO. Calling this function is very easy, the following code illustrated how.

int

}

stdcall

return i+i;

iadd(int i){

int ret = myAdd(iadd,8); printf("results 1: %ld\n", ret )

2.4 Sessions

One particular neat thing is that you can pin certain pieces of Haskell data in memory. This information is not accessible outside of Haskell and thus doesn’t require any marshalling information to be generated. This is handy for when you need to implement session or caching in your haskell code. The following code example should explain it:

type Context α StablePtr IORef α

type

Env Context Int

-- @ Export initEnv::IO Env initEnv newStablePtr newIORef

-- @ Export freeEnv::Env IO freeEnv freeStablePtr

-- @ Export add::Env Int IO add env val do env_value deRefStablePtr env modifyIORef env_value val : $!

-- @ Export = sumPrime sum ::Env IO Int sum env do env_value deRefStablePtr env current readIORef env_value return $ sum current

main::IO Int main do env initEnv add env 1 add env 1

26

hs2lib manual

add env 1 add env 1 val sum env freeEnv env return val

When called, initEnv will return a void which you can’t do any- thing with but pass as arguments to the Haskell functions. This is used extensively in Visual Haskell itself.