Poli Bou » Web Development http://www.polibou.com People. Places. Things. Tue, 22 Jun 2010 18:40:03 +0000 en hourly 1 http://wordpress.org/?v=3.0 Simple and Clean Memcached http://www.polibou.com/2010/02/26/simple-and-clean-memcached/ http://www.polibou.com/2010/02/26/simple-and-clean-memcached/#comments Fri, 26 Feb 2010 22:54:27 +0000 Poli http://www.polibou.com/?p=375 I just recently moved my company’s web infrastructure out from our colo facility and onto Amazon Web Services (mainly EC2). Hopefully I’ll have a writeup on the whole process soon. We started running a couple of heavy ad campaigns and it severely loaded the servers. At the time I had our Memcached setup disabled while debugging the kinks out from the migration.

With the large influx of traffic it was time to setup Memcached again. This gave me a chance to re-examine how it was done. Previously, we would cache various blocks of a given page. Depending on the situation, this can be a good solution if you needed a combination of cached content and dynamic output. However, this method is a little dirty since you have to modify your existing source code.

Here’s a solution that I’ve come up with. Basically, you scrape the entire HTML output and store that in Memcache. I think its very clean and non-invasive. You don’t have to modify the existing code at all.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
<?
////////////////////////////////////////////////////////////////
// index.php
//
// This is an example of memcaching a full static page
////////////////////////////////////////////////////////////////

// This is the page you want to cache
$url = "http://www.mysite/mydynamicpage";

// This function grabs the HTML of the page
function ScrapePage () {
    global $url;

    $ch = curl_init();    // initialize curl handle
    curl_setopt($ch, CURLOPT_URL,$url); // set url to post to
    curl_setopt($ch, CURLOPT_FAILONERROR, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);// allow redirects
    curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); // return into a variable
    curl_setopt($ch, CURLOPT_TIMEOUT, 3); // times out after 4s
    curl_setopt($ch, CURLOPT_POST, 1); // set POST method
    curl_setopt($ch, CURLOPT_POSTFIELDS, ""); // add POST fields
    $result = curl_exec($ch); // run the whole process
    curl_close($ch);
    return $result;
}

// This function stores the HTML into memcache and update if its older than 3 mins
function MemCacheFunction($function_name)
{
    $memcache = new Memcache;
    $memcache->connect('localhost', 11211) or die ("Could not connect");
   
    $memcache_key = md5('sometextkey'.$function_name);
    if ( $memcache_result = $memcache->get($memcache_key) )
    {
        //echo "It Worked!";
        return $memcache_result;
    }  
    //echo "Couldn't Find Key: ".$memcache_key;
    $ret = '';
    $ret .= $function_name();
    $memcache->set($memcache_key, $ret, false, 180);

    return $ret;
}

// A simple condition determines whether to load page from Memcache or process the dynamic page
if ($_COOKIE["loggedin"] == "yes") {
    include ("index_dynamic.php");
} else {
    echo MemCacheFunction("ScrapePage");
}
?>

Typically usage would be for heavy traffic pages such as the homepage of a website. You simply rename your index.php page to something like index_dynamic.php and then use the code below as your original index page. In the code you will have to specify the dynamic page you want to cache and at the bottom a condition to whether load from memcache or the actual page. This part is important. For example, you would only want to show the cache for general traffic such as spiders and visitors who are not logged in.

If you need a good breakdown on installing and configuring Memcached, check this out.

]]> http://www.polibou.com/2010/02/26/simple-and-clean-memcached/feed/ 0 ntsysv – Linux Service Manager http://www.polibou.com/2010/02/02/ntsysv-linux-service-manager/ http://www.polibou.com/2010/02/02/ntsysv-linux-service-manager/#comments Tue, 02 Feb 2010 22:43:22 +0000 Poli http://www.polibou.com/?p=359 I was configuring some linux boxes today and discovered the ntsysv command. This is a great command to see what services are loaded during startup. It pops up a simple GUI where you can enable and and disable services. Just run it at the terminal.

]]> http://www.polibou.com/2010/02/02/ntsysv-linux-service-manager/feed/ 0 Using Symbolic Links http://www.polibou.com/2009/06/11/using-symbolic-links/ http://www.polibou.com/2009/06/11/using-symbolic-links/#comments Thu, 11 Jun 2009 18:26:20 +0000 Poli http://www.polibou.com/?p=288 I can never seem to remember all these Linux commands. I just had to look up how to create a symbolic link. Here it is for reference:

ln -s /path/of/original/file /path/of/linked/file

This will point a file or directory to another file or directory. Why would you want to do this? Well, on our web server we have multiple websites and they share a directory of common include files. So instead of using the server path to get the files, we can just link the local include directory to the global include directory and access the files relatively. It just makes things cleaner.

Here’s how to unlink a symbolic link. Remember not to use a trailing slash. rm will also work.

unlink /path/of/symbolic/link
rm /path/of/symbolic/link
]]> http://www.polibou.com/2009/06/11/using-symbolic-links/feed/ 0 Kill A Running MySQL Query http://www.polibou.com/2009/06/01/kill-a-running-mysql-query/ http://www.polibou.com/2009/06/01/kill-a-running-mysql-query/#comments Mon, 01 Jun 2009 22:40:26 +0000 Poli http://www.polibou.com/?p=183 Somehow you ended up with a query that is taking too long to execute and you need to kill it. First, I’d advise against using this in an UPDATE or INSERT command as I don’t know what will happen. Most likely it’ll will just roll back the DB. For a complex SELECT that just got out of hand, this should be safe.

  1. Log into MySQL from your Linux console.
  2. Show your running queries and find the ID with a long run time. I prefer to check this first in mytop to get a real time view and to see the actual query.
  3. Identify the transaction ID.
  4. Kill it.
  5. Exit and you’re done.

[root@server /]# mysql -u root -p
Enter password: "your password"

mysql>
mysql>  show processlist;
+---------+---------+-------------------+-----------+---------+------+-------+------------------+
| Id      | User    | Host              | db        | Command | Time | State | Info             |
+---------+---------+-------------------+-----------+---------+------+-------+------------------+
| 7614534 | root    | localhost         | NULL      | Query   |    0 | NULL  | SHOW Processlist |
| 7614557 | root    | 10.10.0.111:53626 | myDB      | Query   |  9999|       | NULL             |
| 7614641 | root    | 10.10.0.111:53721 | myDB      | Sleep   |    5 |       | NULL             |
+---------+---------+-------------------+-----------+---------+------+-------+------------------+
3 rows in set (0.02 sec)

mysql> kill 7614557;
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye
[root@server /]#
]]> http://www.polibou.com/2009/06/01/kill-a-running-mysql-query/feed/ 1 Creating High Quality Stock Chart With Flash http://www.polibou.com/2009/03/20/creating-high-quality-historical-stock-chart-with-flash/ http://www.polibou.com/2009/03/20/creating-high-quality-historical-stock-chart-with-flash/#comments Fri, 20 Mar 2009 11:10:27 +0000 Poli http://polibou.com/?p=48

Working at WallSt I’ve delt with numerous data providers for our financial information. Data feeds are not cheap especially when you have to deal with individual exchange fees. Often times when selecting a provider you are limited by their charts and graphs. You can usually get ticker information in XML, but graphical data is usually restricted to modules or API calls for an image.

One solution is to obtain a historical feed and develop a charting system, but this can be expensive and time consuming. So how can we do this cost effectively and quickly?

Thank you Yahoo. Yahoo provides historical quotes in spreadsheet (CSV – comma separated values) format which can be easily parsed. I have no idea how they can provide this freely. But I won’t question it :) . Looking at this service, I see that it has changed many times and there are / has been various methods for obtaining their CSV data. So that’s a word for caution should anything change in the future.

Step 1. How to obtain historical quotes:

You can get the data by using the URL below.

http://ichart.finance.yahoo.com/table.csv?s=YHOO

This will provide you with all the EOD price of a company since its IPO. The method above uses the CSV output on a webpage. Yahoo has other URLs that output a downloadable files. Those work also and can be parsed just as easily.

If you want to be able to select a specific date range you can do it in this format:

http://ichart.finance.yahoo.com/table.csv?s=YHOO&a=0&b=1&c=2008&d=11&e=31&f=2008&g=d

Here I’m selecting data just for the year 2008 from 01/01/08 – 12/31/08.

Let’s break down the URL string and take a look at each parameter.

s = The requested ticker symbol.
a = The starting month in two digit format. (ie: “00″ = January, “01″ = February, etc.)
b = The starting day. (ie: 1, 2, … 30, 31)
c = The starting year in four digit format (ie: “2000″)
d = The ending month in two digit format. (ie: “00″ = January, “01″ = February, etc.)
e = The ending day. (ie: 1, 2, … 30, 31)
f = The ending year in four digit format (ie: “2000″)
g = The data interval (ie: “d” = daily, “w” = weekly, ‘m’ = monthly. The default is ‘daily’ if you ignore this parameter.)

Step 2. Retrieving and Parsing the Data

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<?
////////////////////////////////////////////////////////////////
// data.php
//
// This file will dynamiclly output the CSV locally on your  
// server to overcome the security restriction in Flash .
////////////////////////////////////////////////////////////////

function ScrapePage ($url) {
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $fileoutput = curl_exec($ch);      
    curl_close($ch);
    return $fileoutput;
}

$symbol = $_REQUEST["symbol"];

$output = ScrapePage("http://ichart.finance.yahoo.com/table.csv?s=".$symbol);
$output = str_replace("Date,Open,High,Low,Close,Volume,Adj Close\n", "", $output);

echo $output;
?>

I’m using a simple curl function to get the page. Fopen, wget or whatever your weapon of choice should work too. Since we only need the data, I’m removing the column header with a str_replace. The graphing method I will use will take the data in this format so there will be no parsing or manipulating invovlved. That’s all we need to do for this part.

Step 3. Graphing the Data

Building a full fledge charting system in Flash is quite a bit of work. Adobe has simplified this with Flex, which has built in charting support. It’s a great way to go if you have the knowledge and resources to build some very customized. But I’m gonna take this a few more steps down easy street and use a pre compiled flash suite to do the charting.

I will be using charts from amCharts.com. Not just because its free and easy, these are some really well made charts and they look really good. They have a variety of chart types.

Their stock suite can be downloaded here: http://www.amcharts.com/download

I will be using the event chart type in this example. I picked this one because it is the most familiar layout for stocks. It has the standard line chart but also includes the ability to add events. Which for a stock chart, you can labels for news, splits, dividends, etc.

From the download we will need the following files:
/amstock/amstock.swf – The flash file that displays the graph.
/amstock/swfobject.js – Standard flash embed file.
/examples/events/amstock/amstock_setting.xml – The settings file with output parameters

data.php – This is the file we created above.

Once you have all the files, place them in the same directory. Now we’ll edit the files.

First we’ll modify the amstock_setting.xml file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
<php
////////////////////////////////////////////////////////////////
// settings.php
//
// This file is based amstock_setting.xml (event chart type).
// I've done a bit of editing to clean up the number display
// and added some sample events.
//
// I've also converted it to php so we can dynamiclly pass
// the ticker symbol.
////////////////////////////////////////////////////////////////
?>

<?php echo '<?xml version="1.0" encoding="UTF-8"?>'; ?>
<!-- Only the settings with values not equal to defaults are in this file. If you want to see the
full list of available settings, check the amstock_settings.xml file in the amstock folder. -->
<settings>
    <margins>0</margins>
    <redraw>true</redraw>
    <number_format>  
        <separator>,</separator>
        <decimal_separator>.</decimal_separator>
        <thousand_separator>,</thousand_separator>
        <letters>
            <letter number="1000">K</letter>
            <letter number="1000000">M</letter>
            <letter number="1000000000">B</letter>
        </letters>      
    </number_format>
    <data_sets>
        <data_set did="0">
            <title>East Stock</title>
            <short>ES</short>
            <color>7f8da9</color>
            <file_name>/post-content/flashchart/data.php?symbol=<?php echo $_POST["symbol"]; ?></file_name>
            <csv>
                <reverse>true</reverse>
                <date_format>YYYY-MM-DD</date_format>
                <separator>,</separator>
                <decimal_separator>.</decimal_separator>
                <columns>
                    <column>date</column>
                    <column>open</column>
                    <column>high</column>
                    <column>low</column>
                    <column>close</column>
                    <column>volume</column>
                    <column>adj close</column>
                </columns>
            </csv>
            <events>
                <event>
                    <date>2007-12-25</date>
                    <letter>A</letter>    
                    <description><![CDATA[Christmas Day]]></description>
                </event>
                <event>
                    <date>2008-01-01</date>
                    <letter>B</letter>    
                    <description><![CDATA[New Years Day]]></description>
                </event>
                <event>
                    <date>2008-12-25</date>
                    <letter>C</letter>    
                    <description><![CDATA[Christmas Day]]></description>
                </event>
                <event>
                    <date>2009-01-01</date>
                    <letter>D</letter>    
                    <description><![CDATA[New Years Day]]></description>
                </event>
                <event>
                    <chart_id>second</chart_id>
                    <graph_id>volume</graph_id>
                    <date>2004-05-12</date>
                    <bullet>pin</bullet>
                    <description><![CDATA[Split 2:1]]></description>
                    <axis>true</axis>
                    <size>12</size>
                </event>  
                <event>
                    <chart_id>second</chart_id>
                    <graph_id>volume</graph_id>
                    <date>2009-03-19</date>
                    <bullet>pin</bullet>
                    <description><![CDATA[Fake Dividends]]></description>
                    <axis>true</axis>
                    <size>12</size>
                </event>  
            </events>      
        </data_set>
    </data_sets>
    <charts>
        <chart cid="first">
            <height>60</height>
            <title></title>
            <border_color>#CCCCCC</border_color>
            <border_alpha>100</border_alpha>
            <values>
                <x>
                    <bg_color>EEEEEE</bg_color>
                </x>  
            </values>
            <legend>
                <show_date>true</show_date>
            </legend>
            <column_width>100</column_width>
            <events>
                <color>fac622</color>
            </events>
            <graphs>
                <graph gid="close">
                <data_sources>
                <close>close</close>
                <high>high</high>
                <low>low</low>
                </data_sources>
                <bullet>round_outline</bullet>
                <legend>
                <date key="false" title="false"><![CDATA[<b>Price:</b> {close}    <b>High: </b>{high}   <b>Low: </b>{low}]]></date>
                <period key="false" title="false"><![CDATA[<b>Price:</b> {close}   <b>High: </b>{high}   <b>Low: </b>{low}]]></period>
                </legend>        
                </graph>           
            </graphs>
        </chart>  
        <chart cid="second">
            <height>30</height>
            <title></title>        
            <border_color>#CCCCCC</border_color>
            <border_alpha>100</border_alpha>
            <grid>
                <y_left>
                    <approx_count>3</approx_count>
                </y_left>
            </grid>
            <values>
                <x>
                    <enabled>false</enabled>
                </x>
            </values>
            <legend>
                <show_date>true</show_date>
            </legend>
            <column_width>0</column_width>
            <events>
                <color>db4c3c</color>
            </events>
            <graphs>
                <graph gid="volume">
                    <type>column</type>
                    <data_sources>
                    <close>volume</close>
                    </data_sources>
                    <period_value>average</period_value>
                    <alpha>100</alpha>
                    <fill_alpha>0</fill_alpha>
                    <legend>
                        <date key="false" title="false"><![CDATA[<b>Volume:</b> {close}]]></date>
                        <period key="false" title="false"><![CDATA[<b>Volume:</b> {close}]]></period>
                    </legend>        
                </graph>           
            </graphs>
        </chart>    
    </charts>
    <date_formats>
        <events>DD month YYYY</events>
    </date_formats>
    <data_set_selector>
        <enabled>false</enabled>
    </data_set_selector>
    <period_selector>
        <periods>      
            <period type="DD" count="10">10D</period>
            <period type="MM" count="1">1M</period>
            <period type="MM" count="3">3M</period>
            <period selected="true" type="YYYY" count="1">1Y</period>
            <period type="YTD" count="0">YTD</period>
            <period type="MAX">MAX</period>
        </periods>
        <periods_title>Zoom:</periods_title>
        <custom_period_title>Custom period:</custom_period_title>
    </period_selector>
    <header>
        <enabled>false</enabled>
    </header>
    <scroller>
        <graph_data_source>close</graph_data_source>
        <resize_button_style>dragger</resize_button_style>
        <playback>
            <enabled>true</enabled>
            <speed>3</speed>
        </playback>
    </scroller>
</settings>

What I did here was cleaned up some of the XML and added some sample events. I’ve also changed the extension to php since we will be dynamically passing the ticker symbol to the data.php file.

Next we put everything together to display the graph:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
<?
////////////////////////////////////////////////////////////////
// index.php
//
// Displays the graph and takes a requested symbol
////////////////////////////////////////////////////////////////
?>

<?php
$symbol = $_REQUEST["symbol"];
if (!$symbol) {
    // if no symbol is inputted defaults to yahoo
    $symbol = "YHOO";
}
?>

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Flash Chart</title>
</head>
<body bgcolor="#FFFFFF">
<!-- saved from url=(0013)about:internet -->
<!-- amstock script-->
  <script type="text/javascript" src="swfobject.js"></script>
    <div id="flashcontent">
        <strong>You need to upgrade your Flash Player</strong>
    </div>

    <script type="text/javascript">
        // <![CDATA[       
        var so = new SWFObject("amstock.swf", "amstock", "550", "300", "8", "#FFFFFF");
        so.addVariable("path", "");
        so.addVariable("settings_file", encodeURIComponent("settings.php?symbol=<?php echo $symbol; ?>"));
        so.write("flashcontent");
        // ]]>
    </script>
<!-- end of amstock script -->

<br>

<form method="post" action="<?php echo $_SERVER['PHP_SELF']; ?>">
Enter Symbol: <input name="symbol" type="text" value="<?php echo $symbol; ?>" />
&nbsp;<input name="submit" type="submit" value="Go" />
</form>
</body>
</html>

The form passes our ticker symbol to the setting file which will dynamiclly call the data file with the appropriate symbol.

So there you have it…a nice looking flash graph with historical data. I’ve left out some error checking for the bare minimum to get this done. You can use this technique for other series data. There is one hiccup to all this. It lacks intraday data. That’s pretty vital information when dealing with stock price. However its just a matter of obtaining a feed. Unfortunately, I haven’t been able to find a free source for this data. This would surely seal the deal in creating a no cost flash stock chart that would rival Yahoo and Google.

]]> http://www.polibou.com/2009/03/20/creating-high-quality-historical-stock-chart-with-flash/feed/ 19 Using IP address to Geo-locate http://www.polibou.com/2009/03/16/using-ip-address-to-geo-locate/ http://www.polibou.com/2009/03/16/using-ip-address-to-geo-locate/#comments Mon, 16 Mar 2009 23:42:56 +0000 Poli http://polibou.com/?p=31 The topic of geo-targeting content has come up from time to time here at work but I’ve never taken the time to really implement it. So today I’ve been searching around for a solution and notice that it usually entails purchasing an IP database and a subscription for updates.

The idea of paying for something before testing out the implementation never sits well for me. So I was surprise to find www.hostip.info Community GeoTarget IP Project. Much like open source, everyone contributes to keep the database updated. Best of all, the API queries are free.

Here’s a sample of the API for a simple GET:

http://api.hostip.info/country.php
  US

http://api.hostip.info/get_html.php?ip=12.215.42.19
  Country: UNITED STATES (US)
  City: Sugar Grove, IL

http://api.hostip.info/get_html.php?ip=12.215.42.19&position=true
  Country: UNITED STATES (US)
  City: Sugar Grove, IL
  Latitude: 41.7696
  Longitude: -88.4588

http://api.hostip.info/?ip=12.215.42.19
  [use the URL above for an example - XML too long to paste below]

Use the API with a wget or a curl operation and you’re set.

]]> http://www.polibou.com/2009/03/16/using-ip-address-to-geo-locate/feed/ 2