sorting .csv files

cat input.csv | sort -ug -t',' > output.csv worked better than sort -n option, which I had been using.

Batch downloading SDSS image cutouts

I remember I've done that before, but couldn't find the script. Here is one I quickly put together, based on Min-Su Shin's code.

import os
import db
from utils import *

ids = db.dbUtils.getFromDB('id', dbDir+'MyDB.sqlite', 'mothersample')

for id in ids:
  id = str(id)
  ra = db.dbUtils.getFromDB('ra', dbDir+'MyDB.sqlite', 'mothersample', ' where id = '+id)[0]
  dec = db.dbUtils.getFromDB('dec', dbDir+'MyDB.sqlite', 'mothersample', ' where id = '+id)[0]
  name = "data/cutouts/"+id+".png"
  command = str("wget 'http://casjobs.sdss.org/ImgCutoutDR7/getjpeg.aspx?ra="+str(ra)+"&dec="+str(dec)+"&scale=0.5020920&width=150&height=150&opt=GS&query=&Grid=on&PhotoObjs=on' -O "+str(name)  )
  os.system(command)

Handy grep options: show filename

grep -H or grep -l for filenames only (no matching lines shown).

ds9: an alias for multi-extension data cubes

Since I chronically cannot remember the command-line options to ds9, I added this to my ~/.bashrc: alias meds9=' -scale mode 99.5 -cmap Heat -medatacube'

Scipy: installing from source

My laptop Linux is old and rotten, so I compile newer versions of Scipy from source. It's being said that compiling blas and lapack libraries is notoriously difficult, but it was not, using this guide (with minor filename changes).

awk: making a LaTeX table from a csv file

I found this while digging through my master's notes, potentialy very useful. awk ' {print $1," & ", $2, " & ", $3, " & ", $4, " & ", $13, " & ", $14} ' galaxies_Cat.txt > table.txt

location of Python module file

http://stackoverflow.com/questions/269795/how-do-i-find-the-location-of-python-module-sources

awk: removing lines with duplicate beginnings from a file

awk -F, '!_[$1]++' inputFile > outputFile

awk one-liner

I've been using this for a long time, as most of data I use still comes in unruly, mis-formatted csv files.
awk 'BEGIN {FS =","}; {print $1, $6, $7, $8, $9}' catalogue.csv > cat.csv

Python: script to zip multiple files by filename, flatten structure

A quick script to zip multiple images from different directories.

zip: flatten directory structure

zip -j zipfile path/to/file1 /path/to/file2

mailx: email a file from shell

mailx aaa@bb.cc -a attachmentFilename -r return@adress Ctrl+D to send out.

sed: print specific line of file

sed 'n!d' filename.csv, here n is the line number.

Change the login shell

> chsh
> /bin/bash

ls -- sorting files by size

ls -Slh or ls - Slhr for largest last.

Getting a single file from git stash

Sometimes I do want to keep some local changes, even though stash is such a good way to bury and forget them.
A solution from stackoverflow:

git checkout stash@{0} -- [filename]

opening images with ds9 from a python script

I did it the old-fashioned way (os.system is going to be deprecated):

  import os

  os.system("~/ds9  -zoom 0.3 -scale mode 99.5 -file "+ 

GalaxyParameters.getSDSSUrl(listFile, dataDir, i) +" -file  "+ 

GalaxyParameters.getMaskUrl(listFile, dataDir, simpleFile, i) +" -match frames")

Generating matplotlib plots via ssh on a remote server

If you're running a plotting script via ssh, you're likely to get an error message of this sort:

_tkinter.TclError: no display name and no $DISPLAY environment variable

One possible solution is to use the Agg backend.

Morning coffee read -- ps

ps reference. this command was pretty mysterious to me, with several types of option syntax.

Git -- non-fast-forward updates were rejected

I got the following error message:

To prevent you from losing history, non-fast-forward updates were rejected
Merge the remote changes before pushing again.  See the 'Note about
fast-forwards' section of 'git push --help' for details.

Stackoverflow helped, as usual. As I am the sole developer of my can of worms, I just added the '-f' flag. It's nice to do the last commit of the photometry code which I've been working on for the last 3 months.