Awhile ago I used pip to install python's cssselect.  I'm not sure (should have taken notes!), but it seems like it was difficult, requiring that I get in and install various c libraries.  This time it was effortless.  Here are the instructions that I followed:

http://www.installion.co.uk/ubuntu/saucy/universe/p/python-cssselect/en/install/index.html

Editing Animated Gif duration with GIMP

I made a screencast using Byzanz that went well, except that I had given it a duration that was perhaps 20 seconds too long.  The result was an animated GIF that needed to be truncated.

After a little research, I learned that you can do this with GIMP.

GIMP will import the GIF as a sequence of layers.  And each layer will have a duration as part of its name.  For me, I just had to delete the last layer, which was by far the longest in duration and the problem was solved.  Alternatively, you can edit the duration of the layer by editing its name.

Good argument for seperating pure functions from those that are not so pure

The Opinionated Guide to Python makes a nice argument for keeping functions purely functional (without implicit reliance on global variables and context and without side effects) whenever possible:

http://docs.python-guide.org/en/latest/writing/structure/#object-oriented-programming

Carefully isolating functions with context and side-effects from functions with logic (called pure functions) allow the following benefits:

• Pure functions are deterministic: given a fixed input, the output will always be the same.
• Pure functions are much easier to change or replace if they need to be refactored or optimized.
• Pure functions are easier to test with unit-tests: There is less need for complex context setup and data cleaning afterwards.
• Pure functions are easier to manipulate, decorate, and pass around.

In summary, pure functions are more efficient building blocks than classes and objects for some architectures because they have no context or side-effects.

This speaks to me especially after trying to test some of my methods in Django and seeing the time and pain costs associated with setting up sufficient context to test them.

Gif screencasts

I've been extremely impressed by webpages that have gif screencasts embedded in them. Today, since I committed to write some tutorials, I learned how to make them. I'm using Byzanz, which can be installed on Ubuntu easily: http://askubuntu.com/a/123515 You have to type byzanz-record to get it to run. And you need to specify a file name to record to. Other than that, you just run it.

There are some nice tips on how to limit byzanz to a single window, etc. here: http://primedotprog.com/creating-screencast-gifs-in-linux-ubuntu-14-04-lts/

Two approaches for running Selenium from cron

1) this is easy: give cron a display by adding "export DISPLAY=:0" to your crontab command:
http://blog.markshead.com/392/selenium-no-display-specified/
2) this should work from a server: run firefox headlessly:
http://www.alittlemadness.com/2008/03/05/running-selenium-headless/

Get requests in a ton of different languages

http://rosettacode.org/wiki/Web_scraping

http://rosettacode.org/wiki/HTTP_Request

Testing scripts 2

Testing scripts

Live data into Excel

I may have finally learned the native way to get live numbers into Excel:

http://www.windmill.co.uk/excel/excelweb.html

Can this also be used to publish numbers using Excel?

Simple widgets

I ended up adapting the method at Dr. Nic's site for widgets.

It's fairly simple.  And it can be made nicer by adding in a call for the widget to load asynchronously.  Actually, for my purpose, I needed to generate a whole class of widgets, which meant that the widget itself is generated dynamically.  So I was able to simplify the code by getting rid of the second call to the server to get the data.

Great post on the regular expression module re in python

http://www.dotnetperls.com/re

It takes you through a series of examples on using the re module in practice.  It might be fun to write a miniature book around this topic.

Installing lxml

It turned out to be hugely difficult to install lxml on my Ubuntu lts 14.04 virtual instance.

I think there were several problems.  But the really tough one was pointed out here:

http://stackoverflow.com/questions/16149613/installing-lxml-with-pip-in-virtualenv-ubuntu-12-10-error-command-gcc-failed

Basically, I needed more memory.  So I enlarged the RAM for my digital ocean droplet. 

(I found nice instructions for doing this here.  I don't think I would have had the courage to destroy my own droplet without someone saying that they had done it.  Even so, it was difficult.)

And the error changed from gcc error status 4 to gcc error status 1.  It reported failing to find -lz.

So I added the libz-dev package and ran the following:

sudo python2.7 setup.py clean build --with-cython install

from a git clone of the lxml directory.  So the good news is that now it's installed with cython so maybe it will be faster than it otherwise would have been.

installing missing libraries

Useful post for Ubuntu installations:

http://iambusychangingtheworld.blogspot.com/2013/08/ubuntu-solve-cannot-find-l-errors.html

I quote:

Ubuntu - Solve "cannot find -lxxx" errors

In Linux (Ubuntu), whenever you install a package and you see the following error:

...
/usr/bin/ld: cannot find -lxxx
...


It means that your system is missing the "xxx" lib. In my case, the error occurs when I install MySQL-python package:

/usr/bin/ld: cannot find -lssl  

/usr/bin/ld: cannot find -lcrypto


So, the solution here is install those missing libraries. For my computer, I'just need to install the OpenSSL development lib, the libssl-dev:

sudo apt-get install libssl-dev


And everything will be OK.

The Hitchhiker’s Guide to Python!--the next book

I've been reading a chapter each day from the Django book.  I'm now almost through working through the appendix.  Now, just in the nick of time, I discovered where to go next.

The book is an open-source, ongoing project that is headed by my current python hero,  Kenneth Reitz.

http://docs.python-guide.org/en/latest/

Nice looking cron replacement for django

Nice-looking cron replacement for django:

https://pypi.python.org/pypi/django-periodically/0.3.0

Better deployment

After committing some changes, I've been deploying them using git push.

Then I'd check the pages.  Some (the ones that require changes to Python code) would fail to update and I'd need to restart ssh into the remote repository and restart Gunicorn.  (kill -HUP process_number, where process_number can be found using grep).

Then, if I was smart, I would check the different instantions of web pages to make sure that everything was in order.

Anyway, today I wrote a little script called deploy.  I made it executable, put it in a folder called 'bin' in my home directory, and put it on the path by adding an export command to the bottom of bashrc.

Right now it just chains four commands using &&, so that each one only begins upon the successful completion of the last.


#!/bin/bash
cd /home/doug/workspace/nnumscom/nnums
git push live master && ssh django@[host_ip] 'kill -HUP [process_number]' && ./nopen

nopen is a script that opens a set of about 12 web pages on the remote website so that I can give everything a quick visual.

The nice thing is that now I can take this and improve on it over time.  And, of course, I replace three processes (two of which I might forget) with one.

Web-based http requests

Web-based http requests:

https://www.hurl.it/

Web widget tutorial

alexmarandon.com/articles/web_widget_jquery/

Django default settings

From appendix D of the djangobook:

Default Settings

A Django settings file doesn’t have to define any settings if it doesn’t need to. Each setting has a sensible default value. These defaults live in the file django/conf/global_settings.py.
Here’s the algorithm Django uses in compiling settings:

• Load settings from global_settings.py.
• Load settings from the specified settings file, overriding the global settings as necessary.

Note that a settings file should not import from global_settings, because that’s redundant.

Awesome commands for importing web data to Google Sheets

http://googlesystem.blogspot.com/2007/09/google-spreadsheets-lets-you-import.html

=importXML("URL","XPath expression")
=importXML("http://www.google.com/search?q=live", "//a[@class='l']/@href")



=importHtml(URL, element, index)
=importHTML("http://www.google.com/search?q=define:live", "list", 1)

=importData("URL")
=importData("http://www.nnums.com/api/get/cccccc.n") 


=GoogleReader(URL)  
 

Live updates of charts using Google Sheets

Once you generate live data in Google Sheets, you can use that data to generate live charts.  These charts can themselves be embedded in email html or into a website.

Here's a nice overview:

http://www.christuttle.com/5-ways-to-enhance-websites-with-google-docs/#.VIC2n4_yOE0

Javascript library for aligning decimal points in a table

Apparently this is a bit of a trick in html, but here's a library if you have a table with columns:

https://github.com/ndp/align-column

Now if you don't have a table . . .