Archive for the 'programming' Category

get registered domain in Python and Javascript

April 5, 2011

reg-dom-libs are a set of libraries for C, PHP and Perl to convert an arbitrary domain name to the registered domain name.

  • for simple domains, like or, the task is trivial.
  • for more complicated ones, like or, handling the second level subdomain is a little painful.
  • for exoteric ones, like or or 公司.cn, the problem becomes virtually impossible.
  • after seeing stupid ones, like (believe it or not, the registered domain is!), I gave up finding an elegant algorithm for the problem.
  • a full list of valid registered domain is necessary. luckily, it is available (and nightly updated) here.

inspired by reg-dom-libs, I’ve ported the algorithm to Python and Javascript.
see the tests at the end of each file for an example of the usage.

a demo is availble here.

handling an iframe from the inside

March 30, 2011

if you’ve ever worked with iframes you’ve certainly run into the same origin policy.

from here:

The same origin policy basically limits how scripts and frames on different domains can talk to each other.

this is particularly bad if you want to resize an iframe to accomodate its content, as this is not doable with simple CSS (to the best of my knowledge!). it can be done using window.postMessage, though.

see a demo here.

basically, the content of the iframe fires postMessages to its parent window and the window itself, i.e. the iframe container, handles them. cool…

twill revived

March 11, 2011

after reading how to do automated testing in python, I’ve started using twill, by C. Titus Brown

twill is a simple language that allows users to browse the Web from a command-line interface. With twill, you can navigate through Web sites that use forms, cookies, and most standard Web features.

unfortunately, development on twill seems to have stopped 3 years ago.

I’ve cloned the project on Github, fixed some deprecation warnings and added a cool new feature to inspect headers, as well as html.

testing is more interesting than I thought.

year 2038 bug in MySQL

March 4, 2011

today I hit the “year 2038” bug.

while working on MySQL, I did something like:

mysql> create table y2038 (x timestamp);

mysql> insert into y2038 values ('1900-01-01');
Query OK, 1 row affected, 1 warning (0.04 sec)

mysql> insert into y2038 values ('2050-01-01');
Query OK, 1 row affected, 1 warning (0.04 sec)

mysql> select * from y2038;
| x                   |
| 0000-00-00 00:00:00 |
| 0000-00-00 00:00:00 |
2 row in set (0.00 sec)

mysql> drop table y2038; -- damn

it turned out that MySQL’s TIMESTAMP only accepts dates between 1970-01-01 00:00:01 and 2038-01-19 03:14:07: everything outside this range will be truncated.

at least, it will not be called “Friday the 13th” bug!

use DATETIME instead! and RTFM before doing anything!

tdaemon for python3

February 20, 2011

tdaemon is a neat little utility to do continuous testing in python. unfortunately, the original version only works with python2.

I’ve spent a little time to make it work with python3.


SBCL quicker than C?

December 5, 2010

after reading a nice conversation on comp.lang.lisp I decided to play a little bit with SBCL and Debian’s Shootout Benchmarks.

I considered the spectral norm benchmark and compared the C and SBCL implementations.

I started re-running the tests without any further optimization and the results were similar to the ones reported with C beating SBCL by a factor of 2.
(for the C optimization used, see here).

then, I noticed that there was no (declaim (optimize (speed 3) (safety 0) (space 0))) in the SBCL file!

I’ve added it and rerun the test. here are my results:

gcc 0.15u 0.00s 0.17r
sbcl 0.08u 0.02s 0.21r

gcc 5.60u 0.00s 5.69r
sbcl 5.18u 0.01s 5.41r

gcc 18.81u 0.01s 19.12r
sbcl 17.42u 0.02s 17.76r

the SBCL implementation is actually faster than C!

you can find the code for the tests here.

find hyperlinks with lisp

October 23, 2010

here is a quick-and-dirty script to extract all the unique links from a web page.

it uses cl-ppcre to extract the hyperlinks-like strings from a target string. tested using drakma as web client.

(asdf:oos 'asdf:load-op :drakma)
(asdf:oos 'asdf:load-op :cl-ppcre)

(defparameter *url-re* "href\ *=\ *['\"](\\S+)['\"]")

(defun find-links (str)
  (let ((urls '()))
      (u) (*url-re* str nil :start 0 :sharedp t)
      (pushnew u urls :test #'equalp))
  (nreverse urls)))

  (find-links (drakma:http-request "")))

there are 139 links on this page…

lisp threading example

October 23, 2010

today I wanted to experiment with threads in lisp (sbcl, in particular).

here is a trivial example of how to access a shared resource (a closure, keeping a list of integers), from a set of *nt* threads.

(use-package :sb-thread)

;; number of threads
(defparameter *nt* 10)

;; threads
(defvar *threads* '())

;; mutex to use with shared resource accessed by the threads
(defvar *a-mutex* (make-mutex :name "acc-lock"))

;; shared closure accessed by the threads
(let ((x '()))
  (defun acc (v)
    (with-mutex (*a-mutex*)
                (push v x)))
  (defun get-acc ()

;; print the shared resource
(print (get-acc))

;; wait for all the threads to return
(mapcar #'join-thread
        ;; run the threads
        (loop :for i :below *nt*   
              ;; bind i to x so it is local to the thread
              :collect (let ((x i))
                        (make-thread #'(lambda ()
                                         ;; body of the thread
                                         (sleep (random 2))
                                         (acc x))
                                     :name x))))

;; print the shared resource
(print (get-acc))

it can’t get much simpler than that…

simple wordpress api lisp client

August 4, 2010

I like old things, therefore I started to learn Lisp.

there isn’t a better way to learn than by doing, so I tried to implement a client for the wordpress api. the result, even if not complete yet, it’s so elegant that I’d like to share it.

(require 's-xml-rpc)

(defpackage :wp 
  (:use :cl :cl-user :s-xml-rpc))

(in-package :wp)

(defparameter +interfaces+ 
  '((get-blogs "wp.getUsersBlogs" username password)
    (get-tags "wp.getTags" blog-id username password)
    (get-comment-count "wp.getCommentCount" blog-id username password post-id)
    (get-post-status-list "wp.getPostStatusList" blog-id username password)
    (get-page-status-list "wp.getPageStatusList" blog-id username password)
    (get-page-templates "wp.getPageTemplates" blog-id username password)
    (get-options "wp.getOptions" blog-id username password)
    (delete-comment "wp.deleteComment" blog-id username password comment-id)
    (get-comment-status-list "wp.getCommentStatusList" blog-id username password)
    (get-page "wp.getPage" blog-id page-id username password)
    (get-pages "wp.getPages" blog-id username password)
    (get-page-list "wp.getPageList" blog-id username password)
    (delete-page "wp.deletePage" blog-id username password page-id)
    (get-authors "wp.getAuthors" blog-id username password)
    (get-categories "wp.getCategories" blog-id username password)
    (delete-category "wp.deleteCategory" blog-id username password category-id)
    (suggest-categories "wp.suggestCategories" blog-id username password category max-results)
    (get-comment "wp.getComment" blog-id username password comment-id))
  "Interface definition to WP services")

(defun defunwp (params) 
  (destructuring-bind (name service &rest rest) params
    (setf (fdefinition name) (compile nil `(lambda (host ,@rest &optional (url "/xmlrpc.php")) 
                                             (block ,name 
                                                      (encode-xml-rpc-call ,service ,@rest)
                                                      :host host
                                                      :url url)))))

(mapcar #'(lambda (interface) (export (defunwp interface))) +interfaces+)

it uses the s-xml-rpc package and I’ve tested it with SBCL.

example usage is as follows:

* (wp:get-blogs "" "your-username" "your-password") ; returns the list of blogs for the user
* (wp:get-pages "" your-blog-id "your-username" "your-password") ; returns the list of pages

and so on (see wp::+interfaces+ for function names and required params).


missing Dec_0 and friends from python decimal

July 29, 2010

This fails in py2.4, py2.5.1 and py2.6:

from decimal import Dec_0

But it works in all the other py2.5 subversions!

It seems that exposing Dec_0 (and other similar constants) was considered a bug and “fixed” on Jan 3rd 2009 by renaming it _Dec_0.

Indeed, if I had looked at the source code, I would have seen that Dec_0 was for “internal use only”.

From /usr/lib/python2.5/

##### Useful Constants (internal use only) ################################

# Reusable defaults
Inf = Decimal('Inf')
negInf = Decimal('-Inf')
NaN = Decimal('NaN')
Dec_0 = Decimal(0)
Dec_p1 = Decimal(1)
Dec_n1 = Decimal(-1)

Those definitions are missing in py2.4 and py2.5.1, and renamed in py2.6.

Damn: never use undocumented features!