Why not to use Python's 'eval' in a public service calculator

Here's a fun write-up on python eval security issues:

http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html

Basically, even if you try to restrict access to any and all functions and classes, you can use lambda functions and introspection to get a huge amount of access.

The most fun example in the write-up is

().__class__.__bases__[0].__subclasses__()



This gives a list of all classes instantiated to that point in the program.

cool tool in django: inspectdb

Say you have a legacy database and you want to put django on top of it.  One line will construct python models for the database structure:

python mysite/manage.py inspectdb > mysite/myapp/models.py

You can then clean up the result as explained in chpt 18 of the Djangobook: http://www.djangobook.com/en/2.0/chapter18.html

An address for a specific number on a web page

A fun game to play when you are surfing the web and see something of interest is to right click the object and then choose the option "Inspect Element."  On my desktop in Chrome or in Firefox, what happens when I do this is that developer tools open up and I can look at the element itself.

So that brings up the question: can the address of that element be exported and used?  Can you, instead of linking to a page, link to an element on that page?  If so, can you use that to follow a number or a text field as it changes over time?

The answer, of course, is 'sometimes.'  How quickly the element address breaks depends on the page.

The question of how to reliably get a particular snippet of information from a page is important and I'll be coming back to it as I build nnums. 

Excel to web and back again

The goal here is to learn to take a number in Excel and then automatically post it to a site on the web.

A second goal is to automatically read a number from the web.

This guy used VBA in word to call a Windows library.  So this solution is Windows only, but . . . maybe Excel is mostly Windows only anyway:

http://www.wayne-robinson.com/journal/2009/7/28/wordexcel-http-post.html


 A more general discussion is here:

http://stackoverflow.com/questions/158633/how-can-i-send-an-http-post-request-to-a-server-from-excel-using-vba

From that discussion, one person used Querytables from VBA, which meant his solution worked for both Windows and Mac:
http://numbers.brighterplanet.com/2011/01/06/using-web-services-from-excel/

He also has a little bit to teach about going from the web to Google Docs and back.




But here's a more comprehensive resource if you're looking to learn about Google docs:
http://builtvisible.com/playing-around-with-importxml-in-google-spreadsheets/

Might need to learn about Xpath for this.

It's also possible to link to a specific cell in Excel:
http://support.microsoft.com/KB/197922

Web queries in Excel:
http://www.vertex42.com/News/excel-web-query.html

Glyphicons and CDN

The bootstrap template for a Carousel home page has little arrows called glyphicons that are broken by default.  Unless you use the versions of Bootstrap that are available through the CDN provider.  Here's a description of the problem and solution:

https://stackoverflow.com/questions/18245575/bootstrap-3-unable-to-display-glyphicon-properly/18245741#18245

Configuring nginx to serve static files in django

The setup that I use right now relies on Nginx to handle static files and goes to Gunicorn when there's a need to interface with Python.

I update the files on the server by using 'git push live master,' where 'live' is the name of a repository on the server that has hooks set to update a directory that lives in Django.  That directory includes the static files (css, js) that my pages rely on. 

It is possible to configure Django to automatically copy all static files to a single location. Then all you need to do make sure that that location is the one that your nginx configuration points at.  See the docs here.

In my case, I keep all my static files gathered under a directory named "static" so I just pointed Nginx at that directory.  It's reasonably likely that I'll need to change that pointer at some point, so I'm writing a reminder of what I had to do right here.


First I found the relevant configuration file for nginx.  On my Ubuntu VPS that file was under

/etc/nginx/sites-available/username


where 'username' was the user that I was set up under.  There is also a default configuration file, which would be used in the absence of the more specific username configuration. 

Second, in that file I altered the following address to point directly to my static directory

     location /static {
        alias /home/path/to/my/static/directory;
    }




Third, I refreshed nginx:
sudo nginx -reload


Fourth: There was no fourth step.  That's it!

Using git to update a website

Basically, git is meant for version control.  You have a local repository that has live, working files.  But your remote repository is meant to be bare--without any live files.  This asymmetry is built into the system to avoid having people working on files that then get changed while they are working on them.

But what that means is that if you want to use git to update live files on a server, you have to go through a little extra work.  I say a little, because all you have to do is set up a bare repository that will receive your push.  And then you make a hook so that when that repository is pushed to, it will automatically update the files in your live folder, which will reside elsewhere.  As usual, Digital Ocean has a nice tutorial on the topic:

https://www.digitalocean.com/community/tutorials/how-to-set-up-automatic-deployment-with-git-with-a-vps

Reloading

I have Django running under Gunicorn under NginX on a digital ocean installation that I had set up out of the box.  After upgrading Django to 1.7, I wanted to do a quick test that I hadn't broken anything.

So I tweaked the settings and found that nothing had changed.  Restarted nginX and nothing changed.  Finally I went to Gunicorn.

As mentioned here, you "gracefully reload" using the command

$ kill -HUP masterpid

But that means you have to find out what masterpid is.

Here's one way to find out, given here:

$ pstree -ap|grep gunicorn

look for the number on the top toward the left.

Setting up ssh with public key

https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets

This lets you set up a nicely protected server.

Nice detailed look at Vectors in c++

http://www.codeguru.com/cpp/cpp/cpp_mfc/stl/article.php/c4027/C-Tutorial-A-Beginners-Guide-to-stdvector-Part-1.htm

cron debugging: cat /var/log/syslog

So I wanted to test a cron job and make sure that it was running.  I decided to make it something obvious, so I made my command

/usr/bin/firefox www.google.com

and told it to run every minute.  And I got nothing.  After restarting cron and making sure that the last line in crontab had a newline and everything I could think of, I finally looked at the syslog and learned that it was running.  It just didn't output anywhere I could see it.

So here's another case where it's worth looking at syslog:

cat /var/log/syslog

The cron output isn't visible unless you direct it to a file that you choose and look at that file.

adding commands to manage.py (if you don't like the shell < script.py trick)

Say you want to do something periodically that will affect your django environment (like check email).  You can schedule a task to do that using cron.  But that task needs to be run within your django environment.

One way to do that is to use

python manage.py shell < script.py

(or, if you have your permissions altered via chmod +x, 

./manage.py shell < script.py

)

It works.

If you want to get fancier, you can make a custom command for manage.py:

https://docs.djangoproject.com/en/dev/howto/custom-management-commands/

bootstrap

http://getbootstrap.com/css/

good design

https://medium.com/@erikdkennedy/7-rules-for-creating-gorgeous-ui-part-1-559d4e805cda

https://medium.com/@erikdkennedy/7-rules-for-creating-gorgeous-ui-part-2-430de537ba96

Redis installation and use

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-redis

openshift

https://stackoverflow.com/questions/26871381/deploying-a-local-django-app-using-openshift

Simple site that makes money

http://www.randomamazonproduct.com/

Basically, it's hard not to keep pressing that refresh button.  It's like a window into constant surprise.  It might be worth recreating that project but narrowing the search to be within a particular area for within a particular price range or something.

Unicode fun

Cool tool:

http://www.panix.com/~eli/unicode/convert.cgi?text=Unicode%20fun

Unicode fun
𝖀𝖓𝖎𝖈𝖔𝖉𝖊 𝖋𝖚𝖓
𝕌𝕟𝕚𝕔𝕠𝕕𝕖 𝕗𝕦𝕟
𝑼𝒏𝒊𝒄𝒐𝒅𝒆 𝒇𝒖𝒏 

Ubuntu show desktop

super-ctrl-D

This blog is my memory for now.

Changing fieldnames with django 1.7

It took me a little searching to learn how to do this.  Unlike most migrations, django can't generate name changes automatically.

StackOverflow has a recipe for changing the name of a model and a couple of fields:

http://stackoverflow.com/questions/25091130/django-migration-strategy-for-renaming-a-model-and-relationship-fields

I simplified that recipe for the case when you only want to change a single field name:

https://gist.github.com/dhbradshaw/e2bdeb502b0d0d2acced

database migrations in django 1.7+

http://stackoverflow.com/questions/24311993/how-to-add-a-new-field-to-a-model-with-new-django-migrations

I found migrations a bit sticky in a couple of my applications because I didn't initialize migrations right off.  Anyway, this link has a nice set of instructions that includes the case where you didn't start off initialized.

From the link:

To answer your question, with the new migration introduced in Django 1.7, in order to add a new field to a model you can simply add that field to your model and initialize migrations with ./manage.py makemigrations and then run ./manage.py migrate and the new field will be added to your db. To avoid dealing with errors for your existing models however, you can use the --fake:

1. Initialize migrations for your existing models:
   

   ./manage.py makemigrations myapp

2. Fake migrations for existing models:
   

   ./manage.py migrate --fake myapp

3. Add the new field to myapp.models:
   

   from django.db import models
   
   class MyModel(models.Model):
   ... #existing fields
   newfield = models.CharField(max_length=100) #new field

4. Run makemigrations again (this will add a new migration file in migrations folder that add the newfield to db):
   

   ./manage.py makemigrations myapp

5. Run migrate again:
   

   ./manage.py migrate myapp

Make css work the same across all browsers

Link the following stylesheet to keep CSS the same across all browsers:

<head>
<link rel="stylesheet" src="normalize-css.googlecode.com/svn/trunk/normalize.css">

Thanks Udacity.

Android Intents: really cool

Great "get-started" info and motivation on intents:

http://android-developers.blogspot.com/2012/02/share-with-intents.html


Here's a nice example of taking and sending a screenshot:

https://gist.github.com/nikreiman/2310318

Nice tutorial for http access on Android

http://hmkcode.com/android-internet-connection-using-http-get-httpclient/

Make sure to go to the bottom to look at the asynchronous class.

Redis looks Awesome

http://redis.io/topics/twitter-clone
http://try.redis.io/

There will come a time when I need to build a chrome addon. But not yet.

http://www.sitepoint.com/series/build-a-google-documents-word-count-tool/

csrf and request API

Notes to self on csrf protection in Django:

Django has built-in csrf protection if you use their decorators and form system.  In fact, by default you can't process a POST request without csrf protection.  Unfortunately, that protection acts as a wall against API POST requests not generated by the system.

The solution is simple: two views.  The view that you have to handle a GUI form needs the decorate.

A view that handles a curl or other programmatic request needs to be explicitly absolved of the decorator requirement.  Protection must come from authenticating each request instead of relying on a previous login--but that's not a hassle to an automated system.

from django.views.decorators.csrf import csrf_exempt, csrf_protect

@csrf_exempt
@csrf_protect

Any guess which view needs which decorator?

The form needs csrf protection because it is relying on a previous login.
The API authenticates every time and needs to be csrf_exempt.

Nice guide to string formatting in Python

https://mkaz.com/2012/10/10/python-string-format/

Django restful api

Worth a look:

http://www.django-rest-framework.org/

Chrome developer tool saving

Nice video on a powerful tool.  Most importantly, it shows you how to save the changes that you make.

http://www.youtube.com/watch?v=N8SS-rUEZPg

Also, see

http://stackoverflow.com/questions/6843495/how-to-save-css-changes-of-styles-panel-of-chrome-developer-tools

Nice resource for working with the web from Android.

Working with the web is always chancy because you may not have connectivity.  You don't want anything to hang, so you need your calls to be asynchronous.  Here's a nice library that handles that for you:

http://loopj.com/android-async-http/

The core of django authentication

You can find nice information here:
https://docs.djangoproject.com/en/dev/topics/auth/

and here:
https://docs.djangoproject.com/en/dev/topics/auth/default/

There is a lot to learn.  But for me the core of it is this:

1. authentication comes built in by default with simple user objects that include username, password, and email
2. if you have a username, a password, and an email address, you can set up a new user using

   from django.contrib.auth.models import User
   >>> user = User.objects.create_user('john', 'lennon@thebeatles.com', 'johnpassword') 

3.  you can check the identity of a user in a view that receives a post request using

   from django.contrib.auth import authenticate

       username = request.POST['username']
       password = request.POST['password']
       user = authenticate(username=username, password=password)

Handy curl commands for web requests from the terminal

http://www.codediesel.com/tools/6-essential-curl-commands/

On post requests:
http://superuser.com/questions/149329/what-is-the-curl-command-line-syntax-to-do-a-post-request

Repairing an archived program so that it is executable again

chmod -R +rx

Getting python path associated with default python version

Getting python path associated with default python version:

python -c 'import sys, pprint; pprint.pprint(sys.path)'





Semicolons to delimit lines, pprint to make the output pretty, sys.path to see what will actually be called. 

In my case I learned that my virtual environment version of python is calling libraries outside of that environment.

Celebrating awesomeness: python request library and Uniprot

Uniprot makes is super easy to get protein sequences from a url.
The python request library by the awesome Kenneth Reitz makes it super easy to get the contents of a url.

Here's a screenshot from iPython:

Nice way to install JDK

http://tecadmin.net/install-oracle-java-8-jdk-8-ubuntu-via-ppa/

Lessons learned: don't remove python3 on Ubuntu. Also, how to get a python3 virtual environment with ubuntu 14.4

I was trying to create a virtual environment for python3.4 when I ran into the problem described here:

https://lists.debian.org/debian-python/2014/03/msg00045.html

Apparently, python 3.4 is deliberately broken by default in a way that pops up when you try to set up the virtual environment in the ordinary way.  To quote the source:

The current situation in the Python 3.4 package  

is suboptimal because:

% pyvenv-3.4 /tmp/zz
Error: Command '['/tmp/zz/bin/python3.4',  

'-Im', 'ensurepip', '--upgrade', '--default-pip']'  

returned non-zero exit status 1

Although `virtualenv -p python3.4 /tmp/zz` does work.

The bottom line in the quote above provided the key solution.  But only after my big mistake.

My big mistake: removing python3

apt has me so spoiled that instead of turning to Google with the error message, I tried removing python3 with the plan to then reinstall it.  It was while watching the messages pour out of package after package being removed that I realized my mistake.  I had pulled out a critical part of the modern ubuntu environment and everything built on top of it was being removed.  So I'm backing up all files on that installation and re-installation is coming soon.

nginx, postgresql, django, virtualenv

Nice tutorial here on Digital Ocean:
https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-django-with-postgres-nginx-and-gunicorn

The instructions fell apart a bit once I got to the section configuring postgres.  Basically, when I expected to see a query about a role name it skipped that and started asking me about a password.

So for the configuration of postgres on Ubuntu, I recommend skipping over to https://help.ubuntu.com/community/PostgreSQL  .

After that I went back to the guide.

I found that since I was using python3 and because of a bug in 'pip' where it wants to install things globally, I had to change the

'pip install psycopg2' inside my virtual environment to
'pip3 install psycopg2' to get a local installation with python3.

Next, there is a depracated command in the tutorial that no longer works.
Replace

gunicorn_django --bind yourdomainorip.com:8001

with

 gunicorn myapp.wsgi:application --bind yourdomainorip:8001

Bitbucket

Bit Bucket is an awesome place to start up a project.

You start a repository and it guides you through the steps to sync it with your home repository.  And it gives you wiki and issue tracking built in.  And you can share the project with others and keep it private at the same time.  I love it.