Category Archives: VCS

Git goodies

Oh, Git! The thing that makes our developers life so much easier, yet – as, perhaps, any involved system – having so much left out of attention of the most. It’s far from original to post this XKCD gag here, but it’s just too true:

Git by XKCD

Me myself, I don’t really understand Git. I mean sure, I’ve read few articles on its structure and what acyclic graphs are and how it works etc., but when things go awry – I’m still puzzled.

To help that a little, I started to collect a set of shortcuts and tricks to make frequent problems less hassle. These come in form of scripts – while I could make aliases, I somehow prefer separate scripts as they let you use bash syntax if task is a little more complex than a simple shortcut. What’s also nice is if you name your file (or alias, I suppose) “git-kill-all-humans“, you can then run it as “git kill-all-humans” and even see it in the tab completion for Git commands!

The full set could be found under “git-tools” directory at https://bitbucket.org/hydralien/tools, below are just a couple of the most used ones.

  • git-forget – to use as “git forget .” to loose all the uncommitted changes or “git forget filename” to just a specific file to revert
git checkout -- $@
  • git-origin – to get the remote URL of the repository, useful to share or to clone other repository that resides at the same server (so I just need to change the name)
git config --get remote.origin.url
  • git-out – to see what changes are scheduled to be pushed to origin
git diff origin/`git rev-parse --abbrev-ref HEAD` -- $@
  • git-import – get changeset from a different host, useful when development happens on same repository cloned on many instances – sometimes changes end up on a wrong instance and need to be moved without getting them into the repository
curdir=`pwd`; ssh $1 "cd $curdir ; git diff"|git apply 
  • git-rush – probably the most used command – when the repository is large and there’s many people pushing to it, getting your changes into the origin might be a daunting process. So this one just tries till it’s done – it’s a little overcomplicated for stats reasons (and uses another shortcut, so there’s two of them here), but here it is:
attempt=1 ; time until git repush; do let "attempt++"; echo "No luck, once again..."; done ; echo "Finished in $attempt tries" ; date

and the git-repush:

git pull --rebase && git push

Bottom line here… Git is good – it just takes a few shortcuts to fully appreciate it =)

And of course there’s a hell lot more to automate if required – hooks, configuration etc. etc. etc.

Git grep for Emacs

“git grep” is incredibly useful on large repositories – where regular grep (or awk or whatever) takes minutes to proceed, git grep does the job merely in seconds. Very, very useful. The only thing is, I would really like this working as Emacs search – with files selection, highlighting etc.

Didn’t take too long to find a suitable solution – but took some time to tailor it a little. Basically it’s all taken from https://www.ogre.com/node/447 with some minor adjustments – like, for instance, I do a lot of development in “remote” mode, i.e. I open files from remote hosts via SSH (hence replace-regexp-in-string part). Enjoy!

(defun git-grep (search path)
“git-grep the entire current repo”
(interactive (list (completing-read (concat “Search for (” (current-word) “): “) nil nil nil nil (current-word)) (read-file-name “in directory: ” nil “” t)))
(grep-find (concat “git –no-pager grep -P -n –no-color ”
(shell-quote-argument (if (> (length search) 0) search (or (current-word) “”)))
” ”
(replace-regexp-in-string “/ssh.?:.+:” “” path))))

(provide ‘git-grep)

now if you add this to your .emacs, you can have it launched on some key combination (like Command-c g in my case):

(load-file “~/.emacs.d/git-grep.el”)
(global-set-key (kbd “C-c g”) ‘git-grep)

Git hooks

Git hooks are lovely. Consider automated syntax check before committing changes:

  • git config –global init.templatedir ‘~/.git-templates’
  • mkdir -p ~/.git-templates/hooks
  • vi ~/git-pre-commit-hook.sh and put, for instance, following:


#!/bin/bash

  retcode=0
  for f in $( git diff --cached --name-status|awk '$1 != "R" { print $2 }' ); do
    echo "Veryfying $f..."
    filename=$(basename "$f")
    extension="${filename##*.}"
    if [ "$extension" = "pl" ] || [ "$extension" = "pm" ]; then
      perl -c $f
      lineretcode=$?
      if [ $lineretcode != 0 ]; then
        retcode=$lineretcode
      fi
    fi
  done

  if [ $retcode != 0 ]; then
    echo
    echo "Pre-commit validation failed. Please fix issues or run 'git commit --no-verify' if you're certain in skipping validation step."
    exit 1
  fi

  exit 0

  • chmod a+x ~/git-pre-commit-hook.sh
  • ln -sn ~/git-pre-commit-hook.sh ~/.git-templates/hooks/pre-commit (point of this to have it centralised – on next step global template is copied to repository, so if it’s a symlink – it’s easier to adjust for all later)
  • cd to your repository and run git init there – it’d re-instantiate the repository with copying global hook templates there

That’s it – now you have yourself neat little safeguard. There are better techniques of course – flymake, for one, or some IDE build/check configuration. That works, too – but gets too tricky when you have to patch the code on remote instances. In my case, I have to edit files remotely – and while Emacs has no problem with that (Sublime has plugin for that, too, and you can always add some fuse SSH mount), my flymake settings suck, and I’m not keen to dive into a bloodthirsty piragna pool of its remote execution configuration.

So… git hooks it is.

Project features switch trick (aka Feature Toggle)

OK, so you probably know about feature branches. And DVCS. And CI (see that feature branches article again, there’s also PI idea I wasn’t aware of before). So do I (more or less). What I wasn’t aware of (till recent visit to CEE SEC(R) conference – which was OK, even good at some points) is Feature Toggle idea (sometimes referred as “switches”, “flags” etc.), which is quite useful in stand-alone software model, or if your service is big and world-wide.

The description Martin Fowler provides is pretty complete and verbose (who could have doubted), but still to squeeze the idea: for each feature you can consider a separate chunk of changes – such as a new part in UI that connects to the rest of the interface through a single link, new API method, a switch in data processing algorithm, that kind of stuff – you can do a configuration-level flag that, when given a true value, would enable that feature – or disable otherwise.

I see it useful in many cases – when you can try out new functionality on a limited group of customers but don’t want to build them a separate version, when you want your brand new user profile disguise appear only when customer system is ready for that without blocking other system updates (I know Facebook is using that model to deploy their changes world-wide). This also works quite well with CI idea – you can easily do CI on your main branch, and disable failing features without a need to rollback or quick-fix.

However this Toggling technique (like, well, any other) doesn’t suit all cases. It won’t do for bugfixes; It doesn’t fit small changes, or a patch that alters all occurrences of some method; it’s an overkill for (relatively) small SaS model where releases are easy to fix or rollback. So… Be advised, use wisely. And if you do, don’t forget to remove togglers when the appropriate code reaches stable-prod-level state – otherwise you eventually get yourself another ugly mess of no-one-remembers-why rules.

FTR there’s also this “branch by abstraction” idea – but I think it’s somewhat overcomplicated. I mean , to build an abstraction layer (sort of facade/adapter fusion) and switch internal methods used is alright, but why not just doing those method changes in branches? Although the perspective I see it from is a SaS, and rather tiny one – so there might be valid cases for this one, too.

Move a directory (or a few) to a new repository, with history, in Mercurial

This popped up on my schedule once again today, so I thought I better save it here – JIC.

The goal is simple: project grew big, some parts got a job and moved to LA are big enough to be developed separately. However it’d be better to move them along with their history. Luckily, hg can get you there, with a little to no sweat. Here’s the deal.

First, you need the convert extension enabled.  http://mercurial.selenic.com/wiki/ConvertExtension (that’s where extra details are) says it should be

[extensions]
hgext.convert=

in your ~/.hgrc (or wherever level you keep your Mercurial configuration at), but mine has

[extensions]
convert=

So… go figure yourself.

Then, you need to get a list of what you would like relocated and save it to some file, each line starting with “include” (you can also do “exclude” if you need some inner parts excluded). Like this (I’m using some real name from project so I could share it with colleagues. Names are pretty vague though, so no confidentiality breach involved):

airhydra:uslicer halien$ cat /tmp/targets
include API/REST
include Scheduler
include UI/lib

if you need to rename some directories in new repository, you can do that with adding “rename olddir newdir” (that should go after “include” list) – for instance,

airhydra:uslicer halien$ cat /tmp/targets
include API/REST
include Scheduler
include UI/lib
rename Scheduler Backend/Scheduler

Now, run that convert command, pointing it to a file with targets/actions, source and target (temporary) directory. In my case, source is current directory. Remember: considering on repository size, it could go on for some time and crap in your console like a bloody deranged elephant. Example (crap wiped off):

airhydra:uslicer halien$ hg convert --filemap /tmp/targets ./ /tmp/TempRepo
initializing destination /tmp/TempRepo repository
scanning source...
sorting...
converting...
15855 add data validation
15854 update data validation
15853 add API placeholder
15852 add test data
15851 Initial API skeleton
(...loads of crap...)

Now go to the repository you need those changes at. Here starts the artificial part – I initialized a new repo called NewWorld, but you should create and clone (or, well, just init, if that’s the case) yours in advance. In that repo, pull from the temporary repository you’ve dumped changes into:

airhydra:NewWorld halien$ hg pull -f /tmp/TempRepo
pulling from /tmp/TempRepo
requesting all changes
adding changesets
adding manifests
adding file changes
added 5809 changesets with 7074 changes to 364 files (+166 heads)
(run 'hg heads' to see heads)

Big one, eh? That’s just a fraction. Deserved a move long ago.

If your repository is new, just update to the default branch (or whatever you fancy). If you’re pulling changes into existing one, do a merge. And if the existing repository had some branches named similarly to those available in pulled changeset, you need to resolve conflicts. Which is ugly, so I wish you don’t have to. Or it might be one of the rare cases to justify push -f (given you don’t care much of those branches).

Aaaand… that should be it:

airhydra:NewWorld halien$ ls
API Backend UI
airhydra:NewWorld halien$ hg branches|wc -l
918
airhydra:NewWorld halien$ hg hi|grep changeset:|wc -l
5809

It probably could be done in some more, ehm, (cruelly) sophisticated way through pipes – but I don’t want to bother. This one’s good enough for the case.