Category Archives: development

Turning Reddit RSS into a usable thing, or replacing reddit links with article links

Reddit is a great place to follow various news and updates – I’m using it to keep up to date with programming-related stuff, https://www.reddit.com/r/programming/

Although it could be read directly on the website (I guess), I prefer to do it via RSS aggregator (I use Feedly) – it makes it so much easier to track updates. So adding .rss to the Reddit URL, I get all I need in the aggregator (e.g. https://www.reddit.com/r/programming/.rss)

Well, more than I need, actually – it points me to the Reddit page with comments about the article, and those comments are the very last thing I want to see (if you tried to read them at least once, you’d understand; if not – just try; or trust me).

So doing extra navigation by clicking each article’s subject one more time after Reddit page loads, it finally got me – and I wrote a little proxy page that would replace Reddit comment page link with actual article link in the RSS feed XML.

It’s all here: https://bitbucket.org/hydralien/tools/src/f1196e83ebecca1c4e66c49b8f817e1609527cf6/reddit-streamline/?at=master

There’s a bunch of debug stuff there, and Godaddy-caused library loading nonsense, but the gist is simple – could also be tested at http://hydralien.net/py/direddit/r/programming/.rss

Math is beautiful – it’s a sheer magic to me, but now and then some good people publish approachable articles that allow me get a tiny dash of understanding of how things work.

There’s one such article today – https://innovation.vivint.com/introduction-to-reed-solomon-bc264d0794f8 (and complimentary https://medium.com/@jtolds/joseph-louis-lagrange-and-the-polynomials-499cf0742b39) that I could highly recommend. It’s a fairly simplified overview that provides just a basic idea of a particular error correction technique, but it’s simple yet comprehensive.

In fact it was so fascinating I couldn’t stop myself from giving it a (very short and simple) try. I won’t repeat the articles in any way, just post a code with some comments.

Let’s say we have this code in Python (minor comments inline):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
xs = [ 1.0, # float format to make suure calculation precision is not impacted - it fails badly otherwise
       2.0,
       3.0,
       4.0,
]
 
ys = [ 10, # the values here are random - like in "I made them up". But this is about "any numbers", right?
       42,
       74,
       111,
]
 
# it's a direct representation of what's described in the https://medium.com/@jtolds/joseph-louis-lagrange-and-the-polynomials-499cf0742b39 article
def l0(x):
    return ( (x-xs[1]) * (x-xs[2]) * (x-xs[3]) ) / ( (xs[0] - xs[1]) * (xs[0] - xs[2]) * (xs[0] - xs[3]) )
def l1(x):
    return ( (x-xs[0]) * (x-xs[2]) * (x-xs[3]) ) / ( (xs[1] - xs[0]) * (xs[1] - xs[2]) * (xs[1] - xs[3]) )
def l2(x):
    return ( (x-xs[0]) * (x-xs[1]) * (x-xs[3]) ) / ( (xs[2] - xs[0]) * (xs[2] - xs[1]) * (xs[2] - xs[3]) )
def l3(x):
    return ( (x-xs[0]) * (x-xs[1]) * (x-xs[2]) ) / ( (xs[3] - xs[0]) * (xs[3] - xs[1]) * (xs[3] - xs[2]) )
 
# as well as this
def f(num):
    return ys[0] * l0(num) + ys[1] * l1(num) + ys[2] * l2(num) + ys[3] * l3(num)
 
for x in range(0, 10):
    fx = f(x)
    print "{}: {}".format(x, fx)

And we run it and we get this:

0: -27.0
1: 10.0
2: 42.0
3: 74.0
4: 111.0
5: 158.0
6: 220.0
7: 302.0
8: 409.0
9: 546.0

(you can add more point; you could also use matplotlib to plot them)

So as you can see it reflected the pre-defined 4 points (10, 42, 74 and 111), but also calculated other points on a curve. So let’s say we sent 6 point, but client received only points 1,2,5 and 6 (10, 42, 158, 220).

If we adjust the input values to look like this:

xs = [ 1.0,
       2.0,
       5.0,
       6.0,
]
 
ys = [ 10,
       42,
       158,
       220,
]

and run it again, we’d still get all the values, because 4 values are enough to define the cubic function curve, and these were taken from that very curve:

0: -27.0
1: 10.0
2: 42.0
3: 74.0
4: 111.0
5: 158.0
6: 220.0
7: 302.0
8: 409.0
9: 546.0

Magic, right?

Some extra calculations around it are also available at http://mathonline.wikidot.com/linear-lagrange-interpolating-polynomials

Perl debug with strace

Short “FTR”

Not really specific to perl, but handy anyway.

You can use strace utility to inspect the syscalls (filesystem and network operations are usually of most interest) that a process is making.

here’s e.g. how you can see all network activity for a given process:

strace -p $PID -f -e trace=network -s 10000

Also if you have a stuck process you can check if it is waiting on some filehandle read and then check what that filehandle is using

lsof -p $PID

Also filehandles could be found in /proc/$PID/fd/ – so if you run strace on your process and see e.g.

write(3, "foo\n", 4)

you can check aforementioned lsof | grep $PID and see this

perl       9014 bturchik    3w      REG  253,3        4     26758 /tmp/hung

or

ls -l /proc/9014/fd/3

and see this:

l-wx------ 1 bturchik users 64 Aug  6 09:43 /proc/9014/fd/3 -> /tmp/hung

Git goodies

Oh, Git! The thing that makes our developers life so much easier, yet – as, perhaps, any involved system – having so much left out of attention of the most. It’s far from original to post this XKCD gag here, but it’s just too true:

Git by XKCD

Me myself, I don’t really understand Git. I mean sure, I’ve read few articles on its structure and what acyclic graphs are and how it works etc., but when things go awry – I’m still puzzled.

To help that a little, I started to collect a set of shortcuts and tricks to make frequent problems less hassle. These come in form of scripts – while I could make aliases, I somehow prefer separate scripts as they let you use bash syntax if task is a little more complex than a simple shortcut. What’s also nice is if you name your file (or alias, I suppose) “git-kill-all-humans“, you can then run it as “git kill-all-humans” and even see it in the tab completion for Git commands!

The full set could be found under “git-tools” directory at https://bitbucket.org/hydralien/tools, below are just a couple of the most used ones.

  • git-forget – to use as “git forget .” to loose all the uncommitted changes or “git forget filename” to just a specific file to revert
git checkout -- $@
  • git-origin – to get the remote URL of the repository, useful to share or to clone other repository that resides at the same server (so I just need to change the name)
git config --get remote.origin.url
  • git-out – to see what changes are scheduled to be pushed to origin
git diff origin/`git rev-parse --abbrev-ref HEAD` -- $@
  • git-import – get changeset from a different host, useful when development happens on same repository cloned on many instances – sometimes changes end up on a wrong instance and need to be moved without getting them into the repository
curdir=`pwd`; ssh $1 "cd $curdir ; git diff"|git apply 
  • git-rush – probably the most used command – when the repository is large and there’s many people pushing to it, getting your changes into the origin might be a daunting process. So this one just tries till it’s done – it’s a little overcomplicated for stats reasons (and uses another shortcut, so there’s two of them here), but here it is:
attempt=1 ; time until git repush; do let "attempt++"; echo "No luck, once again..."; done ; echo "Finished in $attempt tries" ; date

and the git-repush:

git pull --rebase && git push

Bottom line here… Git is good – it just takes a few shortcuts to fully appreciate it =)

And of course there’s a hell lot more to automate if required – hooks, configuration etc. etc. etc.

AnotherTab Chrome extension FTW!

OK, the Google Chrome new tab page extension I’ve been (extremely lazily) developing for quite some time is live now! It’s not much, really – just displaying one bookmarks folder (I use “Bookmarks Bar”) and launchable extensions, but that’s what I use the most myself. Oh, and some Chrome shortcuts – like cookies, passwords etc. – too:

AnotherTab screenshot

It’s on Chrome Web Store – but there’s also a separate page to send around: http://hydralien.net/anothertab/

Oh, and the code is public, too: https://bitbucket.org/hydralien/anothertab

Space Slaves released!

Well, at least these folks:

work_in_progress

are no longer smashing the tripgang.com – not that they did much, but enough to let them off to whatever they fancy under those opaque helmets.

So… it’s been quite a while. Now the aforementioned TripGang hosts these:

  • KML2GPX (convert Google KML mars into GPX or OSMAND-friendly GPX)
  • MapMarks (search and bookmark travel pinpoints)
  • WikiVert (auto-search for all points from WikiTravel-like text/attractions list)

It’s quite curious how things turn out – the initial idea for the resource was completely different (well, who knows, I might get to it some day after all) – but hey, “whatever works”, right? Hopefully these tools (however immature and weak they are) might be useful to someone (and most hopefully to myself).

Well… bon voyage, there’s not much else to say really.

 

Exporting Apple Mail filters to Sieve format

What’s this?

It’s a script (in AppleScript) that goes through all Apple Mail filters and converts them to Sieve filters, so you could go server-side on email filtering

Why?

The answer is, as usual, “because I’m lazy” – I’ve accumulated a fair bunch of Apple Mail filters along the way, and converting them manually wasn’t much fun. And I found no working option to convert it on the Internet.

Code and disclaimer

Code is available at:

https://bitbucket.org/hydralien/tools/src/23bca8d016ef88085c98cb1278174be86dfbba4e/apple/Mail2Sieve.applescript?at=master

feel free to clone, submit your patches etc.

NOTE that this is validated with a quite limited use case – in my filters, I mostly match against subject and sometimes against CC/To, so it definitely has some issues with other fields. Please review / try loading the results first and don’t disable all your Mail filters right away.

Exporting Mail filters

Just run that script – it will ask you if you want to use disabled filters as active (useful to re-export after you have already disabled Mail filters), then if you want to disable Mail filters (useful when you’re certain in your Sieve filterset), and then where to save the results.

Was it fun?

Well… the answer is “look at the code”. On the one hand, writing in AppleScript is quite unlike writing in most of the “conventional” languages – some constructs are very different, some seem more natural, others more awkward. On the other, loops management reminded me of programming Turing machine – I mean, using THIS as “continue”, really?!

So to conclude – I think it was, as any unusual experience is fun in it own (however peculiar) way. And it’s the “proper way” for the case – you deal with official API, not parsing the XML (which I could, because Mail rules are stored in XML files) because there’s no way to foretell where the source would move or how its structure would change eventually. Mail API is way less likely to do so.

Resources

Some Sieve-related resources FTR:

What’s next?

Have a beer!

Hash functions

This is “JFTR” – saving few known simple hash functions for further reference (nothing special, really – it’s a “stash” post).

unsigned int HashRs(const char * str)
{

    static const unsigned int b = 378551;
    unsigned int a = 63689;
    unsigned int hash = 0;

    for(; *str; str++)
    {
        hash = hash * a + (unsigned char)(*str);
        a *= b;
    }

    return hash;

}
unsigned int HashLy(const char * str)
{

    unsigned int hash = 0;

    for(; *str; str++)
        hash = (hash * 1664525) + (unsigned char)(*str) + 1013904223;

    return hash;

}
unsigned int HashRot13(const char * str)
{

    unsigned int hash = 0;

    for(; *str; str++)
    {
        hash += (unsigned char)(*str);
        hash -= (hash << 13) | (hash >> 19);
    }

    return hash;

}

 

Retrieving multi-line sequences from text files

Had some free time and had a need to parse out a number if similar (but not the same) blocks from a log file. There are tools for that – it could be done with a mixture of grep, sed, bash and some arcane magic – but I’m afraid to find the right toolset, learn required keys for each and experiment with their values and combination would take me longer than to just write me a tool. And it was another opportunity to write a few lines in Python, which I don’t do often enough. And I do love text processing.

So I made me a neat little tool that essentially does one simple thing – starts printing the input stream when some (start) trigger is found there, and stops when another (end) occurs. There are some additions – like, print some lines before/after the block, print couns, unique blocks only etc. – but those are glitter, mostly.

Available in a BitBucket repository.

WordPress plugins etc.

I’ve been (quite subconsciously) using WordPress for quite some time now, mostly for my alcoholic beverages blog (it’s in Russian, sorry). Subconsciously because it was the first option GoDaddy offered me a “automated install” blogging platform – and also because I’ve heard the name a number of moons back, so it should’ve been well documented and supported at that point. It’s on PHP, but who cares. I’ve spent years writing PHP code.

So I had this problem: my articles all have a rating (I use Author Post Ratings plugin by Philip Newcomer), but it’s not possible to see all the high-rated articles, nor it is possible to order articles by rating within a category – and this feature made a lot of sense, because when you go to a site with a bunch of reviews, you usually look for the best stuff within some category.

So I gave it a thought and just went and added required functionality – now it’s there on bitbucket, https://bitbucket.org/hydralien/author-post-ratings/src

Turned out writing WordPress plugins is a no-brainer if you need something simple (I started with a post-by-rating list) – you just add directory, create a PHP file with a proper header, and you’re done. Well, after you add your functionality, that is. WordPress has some lovely documentation on that.

It gets trickier if you need to change “internal behavior” – such as category sort order – but documentation helps there as well, there are filter hooks for that.

I guess this is worth a slogan – something like “Better drinking with no hassle” or “Drinking better just got easier”. Or whatever.