P. 1
Programming Documentation

Programming Documentation

|Views: 179|Likes:
Published by Max Dovey
This is a log book for the development of the emotional stock market.
This is a log book for the development of the emotional stock market.

More info:

Categories:Topics, Art & Design
Published by: Max Dovey on May 22, 2011
Copyright:Attribution Non-commercial


Read on Scribd mobile: iPhone, iPad and Android.
download as DOCX, PDF, TXT or read online from Scribd
See more
See less





PROGRAMMING DOCUMENTATION for The emotional stock market By Max Dovey


The document is a log book for my process of developing programs that can stream information from the social website twitter in real time. It contains experiments, code, dialogue, and creative thoughts during the devlopment process.

I saw this process in two stages, creating a code in processing that could communicate with twitter (e.g extract data) and secondly visualize it in real time. There are 3 parts that connect for this real time visualization to work.

Processing is a graphically interface for artists and designers to create computer language. Creatives wanting to work with computers directly (no software) have found it the m ost intuitive way to design programs. I have been aware of Processing for a year but have only scratched the surface, knowing that it was a very powerful tool but afraid of code.

¶Twitter4j is an unofficial library for for the Twitter Api·. This means it provides ways to take the information from Twitter (Twitter API) and use it in other ways (programs made in Processing). It links the commands made in Processing to the Twitter API.

Twitter is a social media network and microblogging site that allows personal updates and instant conversations. It has become a tool for user generated media and is now an infamous communication site.

I want to monitor expressions on twitter that use terms that define individual emotions. I want to be bale to visually present real time use of these terms in a graphically way. 1st stage Getting code that can respond and interact with Twitter
Three Steps to a Tweeting Processing Sketch
1. Set up a Twitter application 2. Turn twitter4j into a Processing library 3. Use the right line of code to authenticate the account

I began by downloading the Twitter4j library and importing it into Processing. This was not that easy My Sketch still refused to work saying that it cannot find the Twitter4j Library. I then posted an example code into the sketch and uploaded into the forum.

hey been using this code to get twitter4j working but keep getting this error saying unexpected character "\" but this character doesn't exist in the code so i don't really get it. it also says: processing.app.debug.RunnerException: unexpected char: '\' at processing.app.Sketch.preprocess(Sketch.java:1352) at processing.app.Sketch.preprocess(Sketch.java:1205) at processing.app.Sketch.build(Sketch.java:1568) at processing.app.Sketch.build(Sketch.java:1553) at processing.app.Editor$DefaultRunHandler.run(Editor.java:1485) at java.lang.Thread.run(Thread.java:680)
any help would be much appreciated thank you.

Copy code

1. 2. 3. 4.

/* Posts a message to a Twitter account when you press the mouse button. Uses Twitter4j, http://twitter4j.org. For more info: http://tinkerlondon.com/now/2010/09/13/oauth -twitter-andprocessing/ 5. Daniel Soltis, September 2010 6. */ 7. import twitter4j.conf.*; 8. import twitter4j.internal.async.*; 9. import twitter4j.internal.org.json.*; 10. import twitter4j.internal.logging.*; 11. import twitter4j.http.*; 12. import twitter4j.api.*; 13. import twitter4j.util.*; 14. import twitter4j.internal.http.*; 15. import twitter4j.*; 16. String msg = ´Automatically posted from Processingµ; 17. //copy and paste these from your application in dev.twitter.com 18. String consumer_key = ´publicconsumerkeyfromtwitterdevpageµ; 19. String consumer_secret = ´secretconsumerkeyfromtwitterdevpageµ; 20. String oauth_token = ´accesstokenforyouraccountµ; 21. String oauth_token_secret = ´secretaccesstokenforyouraccountµ; 22. color bgcolor = color(255); 23. long timer; 24. void setup() { 25. size(640,480); 26. } 27. void draw() { 28. background(bgcolor); 29. if (millis()-timer > 2000) bgcolor = color(255); 30. } 31. void mousePressed() { 32. Twitter twitter = new TwitterFactory().getOAuthAuthorizedInstance ( 33. consumer_key, consumer_secret, 34. new AccessToken( oauth_token, oauth_token_secret) ); 35. try { 36. Status st = twitter.updateStatus(msg + µ µ + second()); 37. println(´Successfully updated the status to [" + st.getText() + "].µ); 38. bgcolor = color(0,0,255); 39. timer = millis(); 40. } 41. catch (TwitterException e) { 42. println(e.getStatusCode()); 43. }

44. }
1 user has this question

Re: Problems with twitter4j
by Jesse Scott on 12-Apr-2011 03:14 PM

Contributed Library Forum. Re: Problems with twitter4j
by phi.lho on 12-Apr-2011 10:29 PM

Looks like a bug in Processing itself, in the pre -processor. What version of Processing do you use? Maybe you can try and upgrade (or even downgrade). Or just restart the PDE, somebody with a similar issue had success with this step.. .

The geniuses say reboot?
I attempt to re-install different versions of Processing. 14/04/11 processing version 1.2.1 deleted download and test 1.2 same result download and test 1 not work. Seems like the twitter4j library is not getting into processing. No matter what version of processing I use. I try and install an earlier version of Twitter4j library

I managed to import the twitter4j library by using an older version twitter4j2.1.3 following this guide http://arduino.cc/forum/index.php/topic,57488.0.html

Connection made

Once I had Successfully imported the Twitter4j library I could run some simple example sketches made for example ² Tweeting from clicking my mouse


Now I managed to run an app that tweeted when I clicked my mouse ² I got slightly overexcited

Before I quickly over-used the Twitter API I had to register before they blocked me for spamming the site. This authentication for third party clients (tweeting without twitter) only came to practice last summer. Its an act of data security on Twitters pa rt. So I tried to register on the devlopers site for Twitter. http://dev.twitter.com/ There registration page was down, so I tweeted Twitter ²

Twitter Developers then repaired their site and I was able to regi ster as a developer. This means I can create programs to use Twitter information in different ways.

I then slammed in the earlier code that searches twitter. This works, you can set the amount you retrieve for various search terms. I searched 100 happy and 100 sad. See here. Problems with this code is that it wont fetch more than 100 per query.

how to accurately portray the amount of emotional expressions in realtime
I began to encounter my next issue,
Twitter has strict rules and what you do with the data. Below are restrictions on third party use.

Anonymous calls are based on the IP of the host and are permitted 150 requests per hour. This classificati on includes unauthenticated


requests (such as RSS feeds), and authenticated requests to resources that do not require authentication. OAuth calls are permitted 350 requests per hour.

That·s 6 a minute 1 every 10 secs

The search API does not require authentication. I was trying to work out how often I would grab results from twitter to give an accurate portrait. This goes back to how often you can Search Twitters API (350 ph). 1 search every 10 seconds was generating a lot more than 100 expressions using the word Happy or sad. I worked out that if I can do a new search every 3 Seconds instead of the 100 limit per search.

By asking for the Last ID , I can make sure I don·t count the same tweets twice. For example every search begins at the Last ID from the last search and is not limited to 100 results. By only printing and counting the ID numbers I could save myself a lot of trouble. By printing the user, the text and the Id number it

was running slow, this way I just count how many people have used the search term since the last count. The most efficient way to make a bar graph. Once I had made a method for counting the tweets for a specific search term I needed to visualize it. Then I found ¶The Twitter Party Race· by Daniel Soltis.

This was made in May 2010 to track tweets related to the general election. Daniel uses the last Id to gather new tweets with the search term every 3 seconds. Below is some of his code that I have used.

This Code counts each tweet and strings it into its category. There the amount is turned into a percentage relative of the other total tweets. If it goes up 1 percent than the pixel rectangle increases. To see if it would work I tested my own search terms -

1st Draft. Obviously want to play around with the layout a lot more. This is a format that searches and updates every 3 seconds. The percentage helps to put things into a context. It does stop after 60,000 tweets.

Problems ²

Pretty happy but«.
All search terms begin fast and exciting when there is 100 or less tweets. Once you get to 1000 they remain static due to their size. When analyzing emotional data it seems to all become static, meaning there is constant expressions of basic emotional categories on Twitter. I can use more search terms in one label to widen my search, but perhaps all subjective feelings become mass quantified in a constant stream? I have to address this. It makes all personal expressions turn into categorized, quantified null data. Does this fit with what I am trying to convey? The value of social media as a device for expression and its possibility to measure wellbeing. Also this code does not ask for username/ password and other Security details. I have run it for 30 mins without problems. I want to know why it doesn·t require security authorization and is this a risk? I can add more search terms to one label. So happy could search joy, laughter, pleasure, etc. Atleast it counts all the tweets of people using that word. 5000 for ¶happy· takes less than 5 minutes. Need to turn it into an applet to embed online. Need also to create a sketch that prints tweets. This Bar Graph will be core to my economy system. Each trader for each emotion will have their own feed for that word. I would like to make a sketch that pulls out just the text from tweets with that word every 3 seconds.

This information is useful, With this I now need to expand and make an Economy. A system to buy and sell.

Basic economics and The Stock Market
This graph is data that needs to be combined with the audience buying and selling. So a graph that responds to the frequency info and its selling rate. The frequency of the emotion effects the price. This is what shall be shown online. Each vendor can register a sale of an emotional category. (need to make a manual program for creating a collective graph that is broadcast online) Another basic code for streaming tweets is called ¶simplestream·. Each vendors computer will stream this with their search term. Just entered this code to make it stream ¶sad· "1/statuses/filter.json?track=sad" now trying to work out if im going to be extending the API limits by running 4 stream.twitter.com and (twitter.search) The search API does not require authentiction. (bar chart fine)
µThe Streaming API supports both basic and OAuth authentication on stream.twitter.com. For the time being there is no date on which basic authentication will be turned off for the Streaming API so you are free to choose whichever method you wish.¶ API methods which are not directly rate limited are still subject to organic, unpublished limits. This includes actions like publishing status updates, direct messages, follow/unfollow actions, etc . These Twitter Limits are described on our help site.

e.g. no api limit on publishing status updates? So streaming API is fine.


Returns a random sample of all public statuses. The default access level, µSpritzer¶ provides a small proportion of the Firehose, very roughly, 1% of all public statuses. The ³Gardenhose´ access level provides a proportion more suitable for data mining and research applications that desire a larger proportion to be statistically significant sample. Currently Gardenhose returns, very roughly, 10% of all public statuses. Note that these proportions are subject to unannounced adjustment as traffic volume varies. y URL: http://stream.twitter.com/1/statuses/sample.json y Method(s): GET y Parameters: count, delimited y Returns: stream of status element

Specifies keywords to track. Phrases of keywords are specified by a comma-separated list. Queries are subject to Track Limitations, described in Track Limiting and subject to access roles, described in the statuses/filter method. Comma separated keywords and phrases are logical ORs, phrases are logical ANDs. Words within phrases are delimited by spaces. A tweet matches if any phrase matches. A phrase matches if all of the words are present in the tweet. (e.g. µthe twitter¶ is the AND twitter, and µthe,twitter¶ is the OR twitter.). Terms are exact-matched, and also exact-matched ignoring punctuation. Each comma-seperated term may be up to 60 characters long. Exact matching on phrases, that is, keywords with spaces, is not supported. Keywords containing punctuation will only exact match tokens and, other than keywords prefixed by # and @, will tend to never match. Non-space separated languages, such as CJK and Arabic, are currently unsupported as tokenization occurs on whitespace. Other UTF-8 phrases should exact match correctly, but will not substitute similar characters to their least-commondenominator. For all these cases, consider falling back to the Search REST API. Track examples:

will return statuses which contain: TWITTER, twitter, ³Twitter´, twitter., #twitter and @twitter. y Twitter will not return statuses which contain: TwitterTracker and http://www.twitter.com. y helm's-alee will return statuses which contain: helm¶s-alee y helm's-alee will not return statuses which contain: #helm¶s-alee y twitter api,twitter streaming will return statuses such as: ³The Twitter API is awesome´ and ³The twitter streaming deal is fast´. y twitter api,twitter streaming will not return statuses such as: ³I¶m new to Twitter´. y chirp search,chirp streaming will return statuses such as: ³Listening to the @chirp talk on search´, ³I¶m at Chirp talking about search!´, and ³loving this search talk #chirp´. Values: Strings separated by commas. Each string must be between 1 and 60 bytes, inclusive. Methods: statuses/filter Example: Create a file called µtracking¶ that contains, excluding the quotation marks, the phrase: ³track=basketball,football,baseball,footy,soccer´ then execute:
y Twitter curl -d @tracking http://stream.twitter.com/1/statuses/filter.json uAnyTwitterUser:Password.

You will receive JSON updates about various crucial sportsball topics and events.
2 things I need to sort out visually in code -

Search graph ² add extra bar. Stream ² bar to restart when hits other side. (give co-ordinate) If rect x (etc)

Access and Rate Limiting

All accounts may access the statuses/sample and statuses/filter methods at default access levels. Accounts may also be granted broader data access on


i .A t t t i i l t it itt . t t t t itt it , i f i ti f i ti , t t l l . E t t l t i ti t t St i API. S t ti f t t i l t li ti t i t . Excessive connection ttem ts, egardless of success, will result in an automatic ban of the client's IP address. Continually failing connections will result in your IP address being blacklisted from all Twitter access.


I need an account for each streaming filter sketch. http://dev.twitter.com/pages/streaming_api_concepts still might be too crowded for one api to run 1 search enquiry (every 3 seconds) 3 samping statuses/filter 28/04/2011

TWITTER SAYS ITS GOING TO BE OK Still need to print in stream sketch and have bar graph restart. Design the share index graph a bit more. Could use a status request query sketch to print tweets but need another twitter account for that one station.
Max Dovey, May-19 06:29 am (PDT): H ll , f t i t ll ti i ill i 4 .t itt . . t t itt li , if l t i t t IP lli t t t t 44 ll f t i t l t . l i t it li t i t it t it i t l . l t t 24t , it . t l t. M



t li it i t tf . if i i I il t l t i ll ? l tt ti l i ti


sutorius, May-19 10:11 am (PDT): H ll , Whil the esti ted te f 44 eries er hour from our si le IP should e ithi our limits, lease onsider usi ng the Streaming API for all of our needs and not just the sample: http://dev.t itter. om/pages/streaming_api . or example, ou an onnect to a real time stream of all t eets containing a eyword using the statuses/filter method and the track parameter. We hope that you are able to implement your app with the Streaming API, and are sorry for any inconvenience that this may cause you and your team.

Thanks, Twitter API Policy

The streaming Twitter part of the emotional stock market was always going to be hard. Luckily I located a lot of guides online and teachings from people better at computer programming than me. I have customized a bar graph for counting tweets and have also got a stream bar that shows every tweet using that word. Fortunately it was important to remember this was a piece of art, not social science and could therefore be ambiguous. Its more important to give the impression that every tweet is being counted, than doing it.

Bibliography ² (a list of helpful people and websites) Daniel Solitis , Author of the Twitter Party Race http://www.danielsoltis.com/Recent_Work.html Jay Thor ² Canadian creative data artist good guides http://blog.blprnt.com/source-code-tutorial Mark Mcbride ² made the simple twitter stream for processing. http://mccv.github.com/processing-tweet-stream/ Processing forum ² http://forum.processing.org/ Tinker London ² A guide to processing and Twitter

http://tinkerlondon.com/now/2010/09/14/oauth-twitter-andprocessing/ Twitter Developers ² guides for app developers http://dev.twitter.com/ Twitter 4j ² a java application for twitter and processing http://twitter4j.org/ RobotGrrl ² creative hacker http://robotgrrl.com/blog/2011/02/21/simple-processing-twitter/

You're Reading a Free Preview

/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->