lit: Gaussian processes for machine learning

http://www.gaussianprocess.org/gpml/chapters/

lit: The Zeldovich approximation: key to understanding Cosmic Web complexity

Hidding, Shandarin et al. 2013: seems interesting, modelling the structure of the cosmic web.

lit: Non-linear structure formation

Slides from F. van den Bosch, from a talk given at Jerusalem Winter School.
A Gaussian random field is completely specified by either the two-point correlation function ξ(r), or, equivalently, the power spectrum P(k)

lit: Halo assembly bias

slides by F. van den Bosch.

Jerusalem WS notes: first part of student talks

A few notes, without credits.

• supersonic movement of DM and baryons at z > 10 -- effect on 21cm background, global history of 21 cm signal
• sSFR was surprisingly const b/ween 2 < z < 7 -- find rising sSFR with z
• a new generation of submm galaxies: Redhift survey of Herschel-selected starbursts: dusty super-luminous starbursts at z=2
• submm counts: flat IMF in bursts is required to match them, galaxy pairs contribution: look blended in SCUBA, resolved in ALMA: counts can be matched with standart IMF
• halo quenching -- high masses and z, bulge-related quenching: low z, low masses
• R_{vir} much smaller (2-3 times) than the extent of intercluster gas -- it's not the end of a cluster
• red-sequence fast track
• mass threshold of outflows

Jerusalem WS lecture notes: 09. the IMF and the SFR

By M. Krumholz, slides here. I especially liked Mark's lectures, because he wades into murky, difficult topics of the ISM physics that many researchers like to leave out and assume something.

• THE 2 problems
• IMF:
  • determines stellar feedback (more at top-heavy IMF, etc), abundances, stellar masses
  • Observations:
  • Bastian 2010 -- MW MF plot http://arxiv.org/abs/1001.2965 -- universal IMF in different regions
  • Why a typical star is a few 10ths M_{\odot}? Insensitive to SF environment, metallicity, dwarf/spiral galaxy type
  • Andersen 2009: IMF in MCs (brightest HII region in the Local Group). Sabbi 2008: in SMC (0.2 Z_{\odot})
  • Variation in cDs? van Dokkum & Conroy 2010 (unresolved stars, red & dead galaxies, stars formed at z = 2) -- http://www.nature.com/nature/journal/v468/n7326/abs/nature09578.html: "The direct detection of the light of low-mass stars implies that they are very abundant in elliptical galaxies, making up over 80% of the total number of stars and contributing more than 60% of the total stellar mass. We infer that the IMF in massive star-forming galaxies in the early Universe produced many more low-mass stars than the IMF in the Milky Way disk, and was probably slightly steeper than the Salpeter form in the mass range 0.1M_{\odot} to 1M_{\odot}"
  • Peak location: non-isothermality is required, comes either from:
  • galactic properties, e.g. Hopkins 2012
  • local non-isothermality approximation:
  • isothermality broken by star formation: accreting star is brighter than non-accreting star --> Krumholz 2012: the characteristic mass is set by deviation from isothermality due to SF, the characteristic mass (peak location) depends on pressure of the core (lower mass at higher p). Pressure is set by balance vs. gravity (surface density)
  • Slope: universal, probably due to turbulence
• SFR:
  • bathtub model (gas in, gas out) when t_{SF} << t_H
  • Correlation b/ween molecular gas SD and SF on galactic scales
  • sub-galactic scales: cloud mass vs. no of YSOs, IR luminosity vs. amount of gas (Wu 2005)
  • Galaxy metallicity dependence
  • phase dependence
  • SFR is a function of Toomre Q in galaxy. Dobbs 2011 simulation: self-regulation
  • Top-down model is not sufficient
  • Bottom-up model: what matters is the small scales (local SF law)
  • Why is \epsilon_{eff} (gas conversion to stars ratio) so low, ~1%? Federrath & Klessen 2012 --> few % efficiency for turbulent, virialised objects.
  • l_s -- sound length, size scaling
  • metallicity-phase dependence:
  • why most of the gas is atomic? Dissociation by UV, except in areas where UV is shielded by dust or H_2, 'magic number' -- surface density that shields, ~10M_{\odot}/pc^2
  • why SF follows H_2? Photons that are responsible for dissociation of H_2, are the same that heat the gas, so if the gas is shielded, it cools --> H_2
  • extragalactic phase dependence (SMC has a different SF law because of different metallicity). SF is not only self-regulating process, it depends on global galaxy properties

Jerusalem WS lecture notes: 04. Physics of Star-Forming Clouds

By Mark Krumholz, 'the only obstacle between you and the exciting opportunity to combine drinking with jet lag'. Here are the slides.
Observations:

• SF gas is cold: observations in radio, mm, far-IR
• Diffuse gas: emission lines, dust
• SF ISM is mostly molecular
• H_2: proof that nature has a cruel sense of humour:
  no electronic excitation in cold gas, vibration: mid-IR energy, too high. Rotations: H2 has no dipole mode, no J1 -> J0 transitions, the lowest transition is J2 --> J0. J2 state -- 511 K off ground: no H2 molecules emission.
• CO: proof that astronomers are stubborn bastards:
  
  • If density is high, radiation doesn't change energy distribution (Boltzmann, collisions). Else: way fewer excited molecules than excpected, because collisions don't happen often.
  • Brightness temperature
  • Integrated CO intensity is measure of velocity dispersion (=total gravitating mass), if T = const.
  • Intensity --> directly tells the column density (\Sigma) of CO and H2
  • Motion in gas: bulk, non-thermal, highly supersonic
• Gas properties:
  • cold (10K, 100K in starbursts): adiabatic compression, viscous dissipation, EUV ionisation/FUV photoelectric heating, CR/X ray heating, cooling processes: adiabatic expansion, lines. Dynamical timescales. CRs and X rays can penetrate high columns.
  • Isothermal gas -- efficient cooling if the gas is compressed. Equilibrium T ~ 10K, hard to change. CR -- main source of heating.
  • dense (n > 100cm^{-3})
  • very supersonic: magnetic forces are important, extremely turbulent (Re ~ 10^9).
  • linewidth-size relation (\sigma ~ size of the region), power spectrum
  • VT: thermal motion/thermal pressure prevent collapse, Bonnor-Ebert mass: for a given pressure, there is a maximum mass that can be stable against collapse.
  • for GMCs: M_{BE} ~= 10^7 M_{\odot}: that's why stars form in MCs.

Scipy: installing from source

My laptop Linux is old and rotten, so I compile newer versions of Scipy from source. It's being said that compiling blas and lapack libraries is notoriously difficult, but it was not, using this guide (with minor filename changes).

Deploying my inpainting code on other machines

It's quite a mess, since I have limited permissions on each of them, and their environments vary. Also, no git.
required:
wget
python (developed on 2.6.5)
First, I need pyfits:
mkdir python
cd python
wget http://pypi.python.org/packages/source/p/pyfits/pyfits-3.1.tar.gz
Second, pyfits need setuptools:

wget http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg#md5=fe1f997bc722265116870bc7919059ea
I want to install to a custom location (the ./python directory tree), hence I have to create directory trees for both of them:
mkdirhier [ABSOLUTE PATH TO]/python/lib/python2.7/site-packages/
mkdirhier [ABSOLUTE PATH TO]/python/lib64/python2.7/site-packages/

The .bashrc file's PYTHONPATH variable must be adjusted:
vi ~/.bashrc
export PYTHONPATH=[ABSOLUTE PATH TO]/python/lib/python2.7/site-packages:[ABSOLUTE PATH TO]/python/lib64/python2.7/site-packages/
Do not forget to restart bash:
bash
Then, installing setuptools:
sh setuptools-0.6c11-py2.7.egg --prefix='[PATH TO]/python'
python setup.py install --prefix='[PATH TO]/python'

The file structure:
../growth-curve-photometry/
../data/SDSS/u/ #griz
../data/MASKS/
../data/maskFilenames.csv
../data/maskFilenames.csv
../data/SDSS_photo_match.csv
../data/CALIFA_mother_simple.csv
../data/filled_u #griz

I need AstLib as well:
wget http://downloads.sourceforge.net/project/astlib/astlib/0.6.1/astLib-0.6.1.tar.gz?r=http%3A%2F%2Fastlib.sourceforge.net%2F&ts=1348484024&use_mirror=switch
tar -xvzj astLib-0.6.1.tar.gz
cd astLib
python setup.py install --prefix='[PATH TO]/python/'

...and all the hell breaks loose here, because my home is mounted across the network, hence $PYTHONPATH points to a couple of different Python versions. Separate ~.bashrc.$HOSTNAME files worked: http://askubuntu.com/questions/15618/repercussions-to-sharing-bashrc-across-machines-with-dropbox.
Then fill.py should be edited, or even copied to allow several different processes work in different bands and galaxy sample slices.

Getting a single file from git stash

Sometimes I do want to keep some local changes, even though stash is such a good way to bury and forget them.
A solution from stackoverflow:

git checkout stash@{0} -- [filename]

Failed optimisation: numpy.roll(), masked arrays

I've spent yesterday trying to optimise the inpainting code -- it's quite slow, taking up to 10 minutes for 1 galaxy, and with 900 galaxies, I can't really afford that. Or I should get access to more computers, then prepare the working environment, copy files, install Python libraries, screw something up along the way. That's boring and teaches nothing new.
Besides, there's really some room for improvement: those loops and exceptions are ugly and probably avoidable, and looping over array many times could be sped up using some numpy kung-fu.
I wrote a prototype that juggles with the masks, uses some boolean manipulations and numpy.roll(). Near the edges the algorithm reverts to traditional loops over indices.
However, it's still a slow. I ran a profiler: the loops shifting the array take 0.5s each, and so I win nothing in the long run. Array operations are expensive, and when an array is indexed repeatedly within a loop, the code gets slow inevitably. But there's no way I can escape recursion, because I have to pick up a new set of pixels every time.
I still have some ideas, but I want to have some time to analyse the results.

SciPy: spatial binary tree

I just realised that scipy.spatial.KDTree works in _data_ space, not in _indices_ space, which makes total sense now that I think of it. Should go get me some breakfast, as I'm starting to overcomplicate things.

Inpainting, optimisation, nearest neighbours in Numpy

After looking at my ~900 inpainted images and rejecting ~130 of them outright, I'd like to rewrite my inpainting code. It creates weird artifacts or distorted galaxy views under some circumstances, because it iterates _along_ the axes.
My current idea is to do this recursively: first fill (using the same Gaussian kernel) the pixels that have the nearest 3 4 neighbouring pixels (out of 4 possible) available (not masked), then repeat it again, then go to those that have 3 neighbours, rinse and repeat.
A similar, but possibly a better approach would be to include the diagonal neighbours in the calculation as well (those who have a common vertex, but do not lie side by side). Start with those that have 8 neighbours, (...).
Here's some food for thought: http://stackoverflow.com/questions/9132114/indexing-numpy-array-neighbours-efficiently, http://davescience.wordpress.com/2011/12/12/counting-neighbours-in-a-python-numpy-matrix/, http://stackoverflow.com/questions/5551286/filling-gaps-in-a-numpy-array.

Git -- fixing a detached head

'fixing' as in 're-attaching'. There's nothing intrinsically wrong with having a detached head.
I returned from the state of detached head this way.

'In Git commits form a directed acyclic graph where each commit points to one or more parent commits'.

Geek language, you're a thing of beauty.

Inpainting again

I'm running the inpainting script once more: it took 75 iterations to fix one file. That is, all night. I don't know why I haven't done this before, as this unfinished business was frustrating me throughout the holidays.

the end of MICE

I'm about to finish running GALFIT on the HST F606 MICE image (preview here).
I've used GALFIT before, while writing my MSc thesis, but oh, those were faint, redshifted and unremarkable SDSS galaxies. That was simple -- I used 5 components once, because a galaxy had a LINER AGN.
The MICE are monsters. They've apparently already passed through each other once, and have highly disturbed tidal features, clumps of active star formation, dust lanes, you name it. I've never used GALFIT to model spiral arms, too, so that was an interesting experience.
Here are they:

I started simple, first masking out each galaxy and fitting the other in turn, then fitting both simultaneously (that's a better way to do it anyway). After running about 150 models I decided to mask out the dark dust lanes in galaxy A -- I couldn't fit them using truncation functions, as GALFIT crashes (on my 64bit SUSE, at least) while attempting that and I don't think the science goal required this. I have some reservations about the decomposition, as I think the image has some flat-fielding issues, besides, the galaxies take up a large part of the overall image, so I have doubts about the determined sky value as well.
In the end I had 91 free parameter, and I think more might have made sense, but that was past the point of diminishing returns. I stopped at Chi^2 = 1.877, and guess the main reason why the model would fit the image so poorly were the bright and irregular cores of the galaxies -- or possible higher order flat-field curvature. The other objects were masked out.

Here is the model -- at ds9's 99.5 scale and heat colourmap (this is handy: ds9 -scale mode 99.5 -cmap Heat -medatacube ~/workspace/MICE/galfit/subcomps.fits). The long, bright tails are not well visible with these settings, but they weren't the goal.

I am not really happy with the residual picture, as it tells me there are some big, obscured components lurking underneath. But that'll have to do for the moment, as I have other assignments. I've been doing this for other people in the collaboration, so I don't think I can publish the fit logs here, unfortunately.