You are on page 1of 4

6th International Conference on Electrical and Computer Engineering ICECE 2010, 18-20 December 2010, Dhaka, Bangladesh

Design of a Bridge to Randomly Access High Speed Image Sensor Pixels in Embedded Systems
T. H. Khan#1, G. M. Shahabuddin*2, K. Wahid#3

Department of Electrical and Computer Engineering, University of Saskatchewan, Saskatoon, Saskatchewan, Canada 1 3 * Institute of Information and Communication Technology, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh 2 ATmega644 has 4KB internal memory. However, to store a QVGA (320x240) gray-scale (1 byte/pixel) image, 76.8KB memory is required. Image sensors send the image data in row-after-row fashion. As a result, data can only be accessed sequentially. The first row must be read out prior to the second row, and so on. Many image processing algorithms such as Discrete Cosine Transform (DCT), pattern recognition for robotic vision, need to access pixel values in a random access fashion. For initialization of the image sensor, the 2-wire I2C protocol [8] must be implemented in the microcontroller which is an extra overhead. A high speed clock must be provided to the image sensor. For example, Toshiba image sensor needs to feed a 20MHz clock. So, the embedded system must have a high frequency clock generator. Most image sensors support frames-per-second (FPS) from 15 to 500 and power consumption is typically high. However, for some embedded systems, such as wireless capsule endoscopy [9], lesser FPS (2 to 4) is sufficient and reduction of power consumption is very important.

Abstract—Most commercially available image sensors send image data at high speed and pixel values can only be accessed sequentially. Random access of pixels is not possible because pixel values are sent row-after-row. On the other hand, commercial microcontrollers run at slower speed compared to the high data-rate of the image sensors and many embedded system applications need random access of pixel values. Also, commercial microcontrollers do not have sufficient internal memory to store a complete image. In this paper, the design of a novel bridge is proposed to interface high speed image sensors in low power and low speed embedded systems. By using the proposed bridge, the image processor or microcontroller can capture and store an image in the bridge’s internal memory. Then the pixel values can be accessed in random fashion through a parallel memory access interface at any required speed. The bridge can be used in different embedded system applications such as pattern recognition, robotic vision, biomedical imaging etc. The proposed bridge was modelled in Very-high-speed integrated circuit Hardware Description Language (VHDL) and implemented in Field Programmable Gate Array (FPGA) to verify its correctness. The model was also synthesized in different FPGA devices and result shows that the bridge can support data-rate of image sensors up to 252.1 MHz in Xilinx Vertex 5 FPGA. Keywords— buffering, image sensor, microcontroller, protocol, random access

I. INTRODUCTION In the recent years, image sensors have increased in quality and capability and decreased in price, making them desirous to include in small electronic devices and systems. However, these image sensors are difficult to interface with most commercial microcontrollers due to the following reasons: High-speed image sensors produce data at such a high rate that it cannot be processed in real time. As a consequence, most high-speed image sensors are used in recording systems and difficult to use in low power and low speed embedded systems. For an example, Toshiba (TCM8230MD) [4],[7] image sensor give data at the speed of 6 to 12 MHz. However, most microcontrollers such as Atmel’s ATmega644 [6] can operate at maximum 8 MHz without external crystal. So, real time processing of the pixel values is difficult. No buffering is provided inside the image sensors. Microcontrollers have limited internal memory and may not be able to store a complete image frame unless external memory is provided. For an example,

Fig. 1 The proposed bridge between image sensor and processor

To overcome the above mentioned problems, we are proposing a novel bridge. One side of the bridge will be connected with the image sensor and another side will be connected with the image processor or the microcontroller as shown in Fig. 1. By using the proposed bridge, the image processor can easily initialize the image sensor without implementing the complete I2C protocol firmware. Then the image frame can be captured and stored in the bridge’s internal memory. The pixel values can then be accessed by the image processor at a random fashion through a parallel memory access interface at required speed for processing. To reduce the power consumption, the bridge will automatically send sleep mode command to the image sensor after an image is captured. A clock generator is also included in the bridge to feed the clock signal for the image sensor. In [1], [2], and [3], some works on image sensors with random access are presented. They mainly focus on the analog part of the image sensors. However, our work is in the digital side and

978-1-4244-6279-7/10/$26.00 ©2010 IEEE


600 614. The Adr(18:0). shows the interface pins of iBRIDGE and its internal blocks.576 38. Then the data (pixel value) appears at the Data(7:0) bus after the memory access time which is typically 10 to 20 ns. The VD and HD pins indicate the end of frame’ and end of row’ respectively. the FrameReceived pin goes from low to high. II. THE IMAGE SENSOR INTERFACE Most leading commercial image sensors (such as Toshiba. The rest of the paper is organized as follows.different. THE IBRIDGE The proposed bridge is placed in between the image sensor and the image processor or the microcontroller. the address can be calculated using (1). the common image sensor interface is discussed. (6) and Table II. The frequency of DCLK is half or quarter of the frequency of EXTCLK depending on the configuration of the image sensor. The objective of the proposed bridge is to overcome the speed gaps between the available image sensors and microcontrollers. Most image sensors send some invalid blank bytes and invalid rows while capturing a frame. The address of the first byte can be calculated using (2) and the second byte using (3). III. I. (0 ≤ C ≤ W − 1) and at row R where. Table. The image sensor will be configured with the selected frame size and colour and then image data will began to store in the bridge’s memory block. Adr1 = 2 × C + (2 × W × R ) (2) Adr 2 = Adr1 + 1 (3) After the desired pixel values are read. (0 ≤ R ≤ H − 1) . The EXTCLK is the clock input for the image sensor. sleep mode. it is often called as Serial Camera Control Bus (SCCB) interface [5]. 2. A high on the RGB/Gray’ pin will capture colour RGB image of 16 bits-per-pixel (BPP) and a low on the RGB/Gray’ pin will capture gray level image of 8 BPP. 2.200 76. TABLE I FRAME SIZE SELECTION AND MEMORY REQUIREMENTS FrameSize (1:0) 00 01 10 11 Captured Image Size (W x H) pixels 128 x 96 (subQCIF) 160 x 120 (QQVGA) 320 x 240 (QVGA) 640 x 480 (VGA) Memory Requirement (bytes) 8 BPP 16 BPP 12. and wake up mode can be controlled by sending I2C commands to the image sensor. the image capturing process can be started by asserting the INIT pin to high. shows the possible values of the FrameSize(1:0) pins and its corresponding image sizes with memory requirements. We will refer the proposed bridge as iBRIDGE in this paper. we haven’t found any commercially available chip in the market which serves the purpose of the proposed bridge. the functional description of the proposed bridge and its internal blocks are discussed.can be calculated from (4).400 To access a pixel value. Then the required image size and colour must be selected by the FrameSize(0:1) and RGB/Gray’ pins. Section IV discusses the experimental results and finally Section V concludes the paper. After the image capturing process is completed. The RESET is an active low reset signal for the image sensor. the image sensor goes to sleep mode to save power. In addition. at first image processor 451 . it provides buffering for image storage. Adr = R × W + C (1) For RGB(5-6-5) image [7]. 2 can be connected with an image sensor and the right hand side pins can be connected with the general purpose ports of the image processor or the microcontroller. two consecutive bytes need to be read for one pixel. To access a pixel value of a gray scale image at column C where. The time needed to capture a frame.400 153. 2 Block diagram of iBRIDGE (4) TWakeup = I 2C _ WriteComma ndBytes × = 30 × 1 = 75 × 10 − 6 sec 400 KHz 1 f SCL (5) The left hand side pins of Fig. (5). In Section III. Here. In Section II. asserts the RST pin to high and then make it low. Holding the FrameSize(1:0) and RGB/Gray’ pins to the desired logic. The frame size.200 24. Data(7:0) pins can now be used to access any pixel value of the frame at a slower speed and in a random access fashion. colour. and also pixels can be accessed randomly. Fig. W is the image width and H is the image height. TRe q − Re cieved . While the image capturing process is going on. To capture a frame. the address is placed on the Adr(18:0) bus.800 307. I2C protocol. Fig. the process of capturing the next frame with the same configuration can be repeated by asserting ReqFrame pin from low to high. The initialization and configuration of the image sensor is done by the 2-wire (SCL and SDA) I2C protocol. Pixel data bytes are available for sampling at the DOUT(0:7) bus at the positive edge of the DCLK signal. In the context of image sensor. At the time of writing this paper. OmniVision) sends image data using a common standard interface known as digital video port (DVP) parallel output interface[5]. At this time. The pins of the DVP interface are shown at the left side of Fig. Data(7:0) goes to high impedance state. TRe q −Re cieved = TWakeup + TFrameStore Where. clock generator.288 19.

This clock can be generated by dividing the EXTCLK using a mod-n counter. the iBRIDGE was synthesized in Altera DE2 board’s FPGA [11]. the bridge acts as master and the image using COM port. clock generators. It sends information image processor interface of the iBRIDGE namely. When a ReqFrame signal is received. Memory Addressing and Control This block manages the data pins for the image sensor interface and also generates address and control signals for the Memory block of the bridge. D. When VD and HD are both high. To verify the design in real-world hardware. A level converter IC (MAX232) was used sensor acts as the slave device. The I2C interface block takes slave address. Clock Generator This block generates the clock at the EXTCLK pin. After a complete frame is received. The slave address and sub address for different I2C commands may vary with the different manufacturers of the image sensors. The random access memory block of the bridge was connected with the 512 KB SRAM of the DE2 board. it sends the wake up command and the next image data starts to store in the memory. C. A parallel resonant crystal oscillator [10] can be used to generate the clock. 4 shows the Register Transfer Level (RTL) view of the synthesized iBRIDGE. The serially using one line for data (SDA) and one for clock (SCL). I2C Interface commercial image sensor (TCM8230MD) from Toshiba [7] This block is used to generate the I2C protocol bits in and a commercial microcontroller (ATmega644) from Atmel single master mode [8]. The I2C Interface block generates clock at SCL pin having half of the I2C_CLK frequency. 3. the address up-counter is incremented. These addresses can be taken as input at the time of initialization through the Adr(18:0) bus. After a complete frame is received in the bridge’s memory. It implements a 19 bit address-up counter and it is connected with the address bus of the memory. WR’ signal for the memory is generated. The internal blocks of the bridge (except the RAM block) were synthesized in the Cyclone II FPGA. at each negative edge event of DCLK. it sends command to the image sensor to configure it to the required frame size and colour. Different FPGA pins are connected with different on-board components such as 512 KB of SRAM. Sensor Control This block is used to configure and control different modes of the image sensor. Memory requirements of images of different sizes and colour-depth are shown in Table I. In the valid data state. They occupied 201 logic elements (LE) and 123 registers. A. valid image data comes at the DOUT (7:0) bus. Image Size SubQCIF QQVGA QVGA VGA Init Blank Bytes 157 157 157 157 Pixel Bytes/Row (RGB) 256 320 640 1280 Blank Bytes/Row 1304 1240 920 280 Total Row 254 254 254 507 n 2 2 2 1 Different internal blocks of iBRIDGE are briefly described below. the address up-counter is cleared and FrameReceived signal is asserted high. An 800 KHz clock called the I2C_CLK. which must be fed in the image sensor. E. Only the required subset of 452 . 40 general purpose input/output (GPIO) ports etc. The on-board 27MHz clock generator was used as the clock input for the bridge. and registers data as input from the Sensor Control block and then generates I2C protocol bits at its SCL and SDA output pins. is also required for the I2C Interface and the Sensor Control block. A B. microcontroller was connected with a personal computer (PC) For our application. Then using the I2C interface block. Altera DE2 board has a Cyclone II series (EPC2C35F672C) FPGA. Then a wake up command is sent and then the image sensor starts to give image data. this block sends a sleep mode command to reduce the power consumption. it generates the RESET signal for the image sensor and also read the FrameSize(1:0) and RGB/Gray’ values. Depending upon the application. sub address. EXPERIMENTAL RESULTS The proposed bridge was modelled in VHDL and simulated for functional verification. At each positive edge event of DCLK. The process is implemented in a finite state machine (FSM) structure as shown in Fig. different memory size can be chosen. I2C allows communication of data [6] were connected with the image sensor interface and the between I2C devices over two wires.TFrameStore ⎡ InitBlankBytes + ⎤ n = ⎢(PixelBytes / Row + BlankBytes / Row)⎥ × ⎢ ⎥ f DCLK ⎢× TotalRow ⎥ ⎣ ⎦ (6) TABLE II DATA AND BLANK BYTES SENT BY TOSHIBA IMAGE SENSOR the I2C protocol is implemented to reduce the logic usage. Random Access Memory A single port random access memory block is used to store a frame. The DOUT (7:0) is directly connected with the data bus of the memory. which is required for the image sensor to accept I2C commands for the first time. 3 FSM in the Sensor Control block iBRIDGE were assigned with different GPIO ports. IV. The image sensor interface and the image processor interface of Fig. Fig. Then it waits for 2000 EXTCLK cycle. After the INIT signal is received.

Software was written in MS Visual Basic to display the captured images..0] ENABLE INPUT_BUS[7. “Random Addressable 2048x2048 Active Pixel Image Sensor. 2010 D. December 1998 Toshiba Image Sensor. green (G).0] Fig. http://www. Faigel and D. August 1991 D. The overall experimental setup is shown in Fig. Sodini. www.nxp. “UART Crystal Oscillator Design Guide. McGrath.. Wagner. Our future work includes making a more generalized bridge for different types of image sensors and implementation of the bridge in Application Specific Integrated Circuit (ASIC). of DCLK (MHz) 248. XC2VP2FG256 Virtex4.pdf.6 100.0)_web.0] SDA MEM_ADR_BUS[17. and blue (B) values from the pixel data and then shows the pixel. Cave. The PC extracts the red (R).0] REG_DATA_BUS[7.0] BUF (DIRECT) ent_clk_gen:CLK_GEN1 CLK_27_MHZ_IN CLK_27_MHZ_IN EXT_CLK I2C_CLK UB_B LB_B MEM_DATA_BUS[7. III. XC4VLX15SF363 Spartan3.1 MHz in Xilinx Vertex 5 FPGA. that the maximum allowable frequency for DCLK can vary from 100.0] SCL EXT_CLK WR_B OE_B IS_FRAME_RECEIVED CE_B D_CLK_OUT HD_OUT VD_OUT IS_RGB FRAME_SIZE[1.2 252. www.0] D_ADR_BUS[17. Diamand... R.0] CLK HD INIT IS_I2C_CMD_SET_COMPLETED REQ_FRAME RST VD DATA_BUS[7. 2008 [2] 152 146 149 143 151 212 220 226 189 181 [3] [4] The proposed bridge was also synthesized using Synplify Pro® in different Xilinx FPGA devices.0 200.ovt. the design of a novel bridge is proposed to overcome the speed gaps between the available image sensors and microcontrollers.altera.0] IS_DONE SCL SDA CLK INIT IS_DONE IS_FRAME_RECEIVED IS_RGB REQ_FRAME RST FRAME_SIZE[1.” IEEE Journal of Solid-State Circuits.0] SUB_ADR_BUS[7.. Y. No. 33.. Vol 38.0] ent_sensor_ctrl:SENSOR_CTRL1 ent_i2c_cmd_gen:I2C_CMD_GEN1 CLK RST SEND_CMD SLAVE_ADR_BUS[6. and C. Vol.O.0] REG_DATA_BUS[7. no 8. III.. 12.pdf. http://www. 6 Captured full image (left) and randomly accessed pixel image (right) using iBRIDGE REFERENCES Max Freq. at any desired speed.pdf.0] WR_DATA_BUS[7.1 [1] O. The microcontroller was running at 1 MHz..0] OUTPUT_BUS[7.atmel.0] ent_tri_buf:TRI_BUF1 comb~[15. [11] Altera DE2 Board.. 6 shows a full image (left) and a randomly accessed pixel image (right) captured by the microcontroller using iBRIDGE. Then it returns the pixel value to the PC. The synthesis results except the memory block and the clock generator block are shown in Table. a slow and low power microcontroller having less memory capacity..sparkfun. 4 RTL view of the synthesized iBRIDGE to generate the appropriate logic levels to communicate with the PC. Then the microcontroller calculates the memory addresses using (1)..ent_mem_adr_ctrl:MEM_ADR_CTRL D_CLK HD INIT REQ_FRAME RST VD DATA_BUS[7. TABLE III SYNTHESIS RESULTS ON XLINX FPGA Area Utilization Xilinx FPGA Device Virtex2p. Scheffer.. XCV50ECS144 CONCLUSION In this work.0] MEM_ADR[17.. bio-medical imaging etc. Also. it can access the pixel values in a random access fashion through a parallel memory access interface. Dierickx. and Y. B. 5 Experimental setup can capture image without implementing any I2C protocol in it.0] ent_tri_buf:TRI_BUF2 ENABLE INPUT_BUS[7. No. Pecht. V. and (3) and reads the corresponding pixel data from the iBRIDGE’s memory. By using the proposed bridge. October 1997 S. From the user interface of the PC software.. Vol.. We see from Table..” Data Communications Application Note..0] HW_RESET IS_I2C_CMD_SET_COMPLETED SEND_CMD SLAVE_ADR_BUS[6.. http://www. S. Brehmer. D. 2010 453 .0] RD_DATA_BUS[7. 2010 Atmel 8 bit AVR.0] PRC_ADR[17. Ginosar. Fig. “A 256x256 CMOS Imaging Array with Wide Dynamic Range Pixels and ColumnParallel Digital Output. Meynants.0] OUTPUT_BUS[7. XC5VLX330 Registers Look Up Tables (LUT) Fig.. The bridge was implemented in the real-world FPGA to verify its correctness.0] HW_RESET mux_adr:MUX_ADR1 SEL BRG_ADR[17.” IEEE Transactions on Electron Devices. R.. 5. XC3S50TQ144 Virtex E. and G. 2010 OmniVisoin CameraCube. [5] [6] [7] [8] [9] [10] R.0] CE_B (GND) IS_DONE LB_B (GND) OE_B (GND) UB_B (VCC) WR_B ADR_BUS[17.. 2010 The I2C Bus Specification. Synthesis results showed that the bridge can support data rate of image sensors up to 252. Saunders Elsivier.2 MHz to 252. robotic vision.1 MHz.0] SUB_ADR_BUS[7. “A Random Access Photodiode Array for Intelligent Image Capture. Capsule Endoscopy. 10. user can randomly send column (C) and row (R) positions of an image to the microcontroller. G. Its application may include pattern recognition. (2). March 2000.” IEEE Transactions on Electron Devices.3 142. 2010 Toshiba TCM8230MD datasheet...