Radial Charts

Why

I wanted to plot the data that is tightly connected to a time of a day as a circular chart, resembling the clock face – for no particular reason other than checking out how it looks.

Another thing I wanted to do was to play with some relatively modern JS framework.

And then I also got a bit a of a spare time on my hands.

What

So as a result, here’s a code that fetches CSV data from Graphite metrics (or any other CSV file) and displays it on a radial chart (aka Radar chart). It’s crude and unoptimised and inefficient, but it serves the goal of visual representation (see examples section below): https://github.com/hydralien/Radial-Charts

It’s mostly an amCharts showcase – amCharts are amazing, I love them dearly. Check them out if you’re not familiar, there’s seemingly nothing you can’t do with them: https://www.amcharts.com/demos/

Additionally, it also uses Github Actions (https://help.github.com/en/actions) as a CI mechanism – so on every push a build is pushed to a branch and then deployed as Github pages using https://github.com/marketplace/actions/github-pages-action#%EF%B8%8F-publish_branch (there’s many action plugins, this one was just the first one that worked).

Examples

Surprisingly, there’s npt that many public datasets with high-granularity daily data available on the Internet, but here’s at least one:
Bike rentals in NYC, January 1st 2020 vs July 1st 2019 (from https://s3.amazonaws.com/tripdata/index.html):

Bike rentals

Shortcomings

  • it’s not a working tool or a developing project – it’s a fun thing built for no reason
  • it loads a ton of unnecessary libraries, becasue how amCharts library for Vue.js is structured, and also because it’s not optimised for anything
  • <many other things>

Mandelbrot set

I was always fascinated by the Mandelbrot set pictures and always wanted to create a representation myself, but never really got to do it – till a few days ago when I stumbled over an article boiling the code down to a very simple sequence (https://slicker.me/fractals/excel.htm) – so I just took it and made parameters adjustable. Also used it as an exercise in Vue.js, as it was another thing I wanted to try but never did.

It’s a very-very crude solution, but it was a good exercise: http://hydralien.net/mandelbrood.html?pixelSize=4&divisor=50&cxStart=-2&cyStart=-2&boundary=200&precision=0.4&pixelIterations=30

Also the code is here (well, it’s also in the html file, but FTR): https://github.com/hydralien/mandelbrood

Permanently adding SSH private key to OSX keychain

It’s been bugging me for quite a while, but never enough to go and find a solution – until now.

Shamelessly re-posting a perfect guidance from https://apple.stackexchange.com/questions/48502/how-can-i-permanently-add-my-ssh-private-key-to-keychain-so-it-is-automatically:

On OSX, the native ssh-add client has a special argument to save the private key’s passphrase in the OSX keychain, which means that your normal login will unlock it for use with ssh. On OSX Sierra and later, you also need to configure SSH to always use the keychain (see Step 2 below).

Alternatively you can use a key without a passphrase, but if you prefer the security that’s certainly acceptable with this workflow.

Step 1 – Store the key in the keychain

Just do this once:

ssh-add -K ~/.ssh/[your-private-key]

Enter your key passphrase, and you won’t be asked for it again.

(If you’re on a pre-Sierra version of OSX, you’re done, Step 2 is not required.)

Step 2 – Configure SSH to always use the keychain

It seems that OSX Sierra removed the convenient behavior of persisting your keys between logins, and the update to ssh no longer uses the keychain by default. Because of this, you will get prompted to enter the passphrase for a key after you upgrade, and again after each restart.

The solution is fairly simple, and is outlined in this github thread comment. Here’s how you set it up:

  1. Ensure you’ve completed Step 1 above to store the key in the keychain.
  2. If you haven’t already, create an ~/.ssh/config file. In other words, in the .ssh directory in your home dir, make a file called config.
  3. In that .ssh/config file, add the following lines:Host * UseKeychain yes AddKeysToAgent yes IdentityFile ~/.ssh/id_rsa Change ~/.ssh/id_rsa to the actual filename of your private key. If you have other private keys in your ~.ssh directory, also add an IdentityFile line for each of them. For example, I have one additional line that reads IdentityFile ~/.ssh/id_ed25519 for a 2nd private key.The UseKeychain yes is the key part, which tells SSH to look in your OSX keychain for the key passphrase.
  4. That’s it! Next time you load any ssh connection, it will try the private keys you’ve specified, and it will look for their passphrase in the OSX keychain. No passphrase typing required.

Web server in OSX (Apache2, dammit!)

There’s an Apache2 in OSX!!!

Apache2! It’s been awhile since I touched httpd.conf… so many memories… 😉

It’s all described in this tutorial: http://osxdaily.com/2012/09/02/start-apache-web-server-mac-os-x/

It’s dead simple – see/edit /etc/apache2/httpd.conf to find or set DirectoryRoot (mind directory access rights) and run “sudo apachectl start” (stop, restart).

To enable user websites, add following to etc/apache2/users/USERNAME.conf :

<Directory "/Users/USERNAME/Sites/">
Options Indexes Multiviews
AllowOverride AuthConfig Limit
Order allow,deny
Allow from all
</Directory>

and then it’d be served as http://127.0.0.1/~USERNAME/

Also, generating self-signed certificates: https://devcenter.heroku.com/articles/ssl-certificate-self

And adding them to OSX keychain: https://tosbourn.com/getting-os-x-to-trust-self-signed-ssl-certificates/

And you’d need to put them into /private/etc/apache2/ssl/ and (potentially) edit /private/etc/apache2/extra/httpd-ssl.conf and enable (uncomment)

Include /private/etc/apache2/extra/httpd-ssl.conf

and

LoadModule ssl_module libexec/apache2/mod_ssl.so

in /etc/apache2/httpd.conf

and run “apachectl restart” as root.

Champagne for all!

  _
 {_}
 |(|
 |=|
/   \
|.--|
||  |
||  |    .    ' .
|'--|  '     \~~~/
'-=-' \~~~/   \_/
       \_/     Y
        Y     _|_
       _|_

Turning Reddit RSS into a usable thing, or replacing reddit links with article links

Reddit is a great place to follow various news and updates – I’m using it to keep up to date with programming-related stuff, https://www.reddit.com/r/programming/

Although it could be read directly on the website (I guess), I prefer to do it via RSS aggregator (I use Feedly) – it makes it so much easier to track updates. So adding .rss to the Reddit URL, I get all I need in the aggregator (e.g. https://www.reddit.com/r/programming/.rss)

Well, more than I need, actually – it points me to the Reddit page with comments about the article, and those comments are the very last thing I want to see (if you tried to read them at least once, you’d understand; if not – just try; or trust me).

So doing extra navigation by clicking each article’s subject one more time after Reddit page loads, it finally got me – and I wrote a little proxy page that would replace Reddit comment page link with actual article link in the RSS feed XML.

It’s all here: https://bitbucket.org/hydralien/tools/src/f1196e83ebecca1c4e66c49b8f817e1609527cf6/reddit-streamline/?at=master

There’s a bunch of debug stuff there, and Godaddy-caused library loading nonsense, but the gist is simple – could also be tested at http://hydralien.net/py/direddit/r/programming/.rss

Math is beautiful – it’s a sheer magic to me, but now and then some good people publish approachable articles that allow me get a tiny dash of understanding of how things work.

There’s one such article today – https://innovation.vivint.com/introduction-to-reed-solomon-bc264d0794f8 (and complimentary https://medium.com/@jtolds/joseph-louis-lagrange-and-the-polynomials-499cf0742b39) that I could highly recommend. It’s a fairly simplified overview that provides just a basic idea of a particular error correction technique, but it’s simple yet comprehensive.

In fact it was so fascinating I couldn’t stop myself from giving it a (very short and simple) try. I won’t repeat the articles in any way, just post a code with some comments.

Let’s say we have this code in Python (minor comments inline):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
xs = [ 1.0, # float format to make suure calculation precision is not impacted - it fails badly otherwise
       2.0,
       3.0,
       4.0,
]
 
ys = [ 10, # the values here are random - like in "I made them up". But this is about "any numbers", right?
       42,
       74,
       111,
]
 
# it's a direct representation of what's described in the https://medium.com/@jtolds/joseph-louis-lagrange-and-the-polynomials-499cf0742b39 article
def l0(x):
    return ( (x-xs[1]) * (x-xs[2]) * (x-xs[3]) ) / ( (xs[0] - xs[1]) * (xs[0] - xs[2]) * (xs[0] - xs[3]) )
def l1(x):
    return ( (x-xs[0]) * (x-xs[2]) * (x-xs[3]) ) / ( (xs[1] - xs[0]) * (xs[1] - xs[2]) * (xs[1] - xs[3]) )
def l2(x):
    return ( (x-xs[0]) * (x-xs[1]) * (x-xs[3]) ) / ( (xs[2] - xs[0]) * (xs[2] - xs[1]) * (xs[2] - xs[3]) )
def l3(x):
    return ( (x-xs[0]) * (x-xs[1]) * (x-xs[2]) ) / ( (xs[3] - xs[0]) * (xs[3] - xs[1]) * (xs[3] - xs[2]) )
 
# as well as this
def f(num):
    return ys[0] * l0(num) + ys[1] * l1(num) + ys[2] * l2(num) + ys[3] * l3(num)
 
for x in range(0, 10):
    fx = f(x)
    print "{}: {}".format(x, fx)

And we run it and we get this:

0: -27.0
1: 10.0
2: 42.0
3: 74.0
4: 111.0
5: 158.0
6: 220.0
7: 302.0
8: 409.0
9: 546.0

(you can add more point; you could also use matplotlib to plot them)

So as you can see it reflected the pre-defined 4 points (10, 42, 74 and 111), but also calculated other points on a curve. So let’s say we sent 6 point, but client received only points 1,2,5 and 6 (10, 42, 158, 220).

If we adjust the input values to look like this:

xs = [ 1.0,
       2.0,
       5.0,
       6.0,
]
 
ys = [ 10,
       42,
       158,
       220,
]

and run it again, we’d still get all the values, because 4 values are enough to define the cubic function curve, and these were taken from that very curve:

0: -27.0
1: 10.0
2: 42.0
3: 74.0
4: 111.0
5: 158.0
6: 220.0
7: 302.0
8: 409.0
9: 546.0

Magic, right?

Some extra calculations around it are also available at http://mathonline.wikidot.com/linear-lagrange-interpolating-polynomials

Perl debug with strace

Short “FTR”

Not really specific to perl, but handy anyway.

You can use strace utility to inspect the syscalls (filesystem and network operations are usually of most interest) that a process is making.

here’s e.g. how you can see all network activity for a given process:

strace -p $PID -f -e trace=network -s 10000

Also if you have a stuck process you can check if it is waiting on some filehandle read and then check what that filehandle is using

lsof -p $PID

Also filehandles could be found in /proc/$PID/fd/ – so if you run strace on your process and see e.g.

write(3, "foo\n", 4)

you can check aforementioned lsof | grep $PID and see this

perl       9014 bturchik    3w      REG  253,3        4     26758 /tmp/hung

or

ls -l /proc/9014/fd/3

and see this:

l-wx------ 1 bturchik users 64 Aug  6 09:43 /proc/9014/fd/3 -> /tmp/hung

Distributed development keynotes

Attended UX Lausanne for the second time this year – and while it was rather small (which is actually good as you’re better connected to speakers in terms of Q&A and follow-up), it was not solely on UX but a lot on workflow and project development – something that seems to be called Product Experience.

One of the talks was given by a guy from Automattic company (Davide Casali) – first I thought they’re WordPress authors, but turned out they’re (large) contributors and maintainers of WordPress.com – that has a distributed nature, i.e. everyone works remotely. Like, everybody – few colleagues might use same shared space, but even then they’re not in the same team. Now that is definitely not the case for everyone, but nowadays many companies allow at least temporary remote work, thought of sharing key points here – I think they’re quite interesting and some of them could be used in different cases.

So here are their principles:

  • before creating new team (or starting new project), meet live to have “Minimum Viable Discussion” to quickly understand key points, sort out initial discrepancies in domain knowledge and prepare brief plan for the closest next steps
  • transparently share all changes, specs and discussions – “if it’s not transparently shared, it doesn’t exist” (more office-bound sticky-note-flavor version of this proverb was “if it’s not on the wall, it doesn’t exist”)
  • have few communication spaces for different needs:
    • Real-time channel (like Slack or Jabber or, sigh, Facebook) for immediate personal and team communication
    • Team space on some shared documents resource – they, as WordPress-targeted company, use a P2 theme for WordPress that allows posting comment to posts and comments to comments – basically to host a discussion on the subject. Point is to have something that could be subscribed to or viewed by any employee, and that team members would overview daily to see/discuss updates or refer to while developing
    • “Stable” documentation storage for more permanent things like articles or specs of deployed products etc.
  • each team (BTW they also have small teams, 4-5 people of different skills) focuses and collaborates on one thing (project or task)
  • Independent individuals, i.e. everyone maintains own priorities. Tasks are managed with any suitable tracker (they use Trello)
  • “zero waste” in terms of no bureaucracy regarding e.g. permissions or access
  • standup-kind updates are posted to team channel each time team member becomes available – along with overview of the progress, provides an indication when person becomes available
  • live meetups ~4 times a year – 3-5 days of work, discussions and some off-work time together
  • teammates (but not projects) are usually picked from within few timezones from one another to aid live communication
  • everyone can follow any other team by subscribing to their shared documentation channel

This is all “JFTR”, but some could really consider utilising part of these practices for daily work and communication improvements.

Also another reference to keep, https://www.helpscout.net/blog/agile-remote-teams/