How I do it

In this series I explain how I do certain, mostly technical, stuff. Basically it is written for the audience that is just like me (in other words: this is just my public external memory ;) ). For those, who like me, forgets about the details when doing this somewhat more advanced server stuff, but is capable of throwing in some commands at the command line and cares about doing stuff in the right way.

*nix: find the largest files/directories within a directory

An article, posted 3 months ago filed in , , , , , & .

Every now and then I’m searching for this little snippet in my notes using NotationalVelocity (or currently actually a fork):

du -hsx * | sort -rh | head -100

It’s a variation of a snippet I found somewhere, but hardly invested any time in understanding what it actually does. Let’s decompose, from head to taildu.

head -100

head simply limits the results to a maximum of 100 lines. Not much more to explain here

sort

sort sorts. by default it sorts the files by filename, but adding ‘-h’ to it allows it to sort by “human readable numbers” (e.g. 5M > 6K); if ‘-n’ would be added as option 6K would be > 5M. The ‘-r’ options reverses the sort wich is by default ascending.

du

du by defaults crawls a directory recursively for all files. passing '-s' tells it to sum the values of files within directories. the '-x' option is used to n…

Continue reading...

Becoming your own local certificate authority (and issue your first certificate)

An article, posted 9 months ago filed in , , , , & .

It has been quite some time ago, but here is another 'how i do it' article :)

If, by 'accident' you have, like me, chosen projectname.dev for your local development as a convention, and you want to continue using this convention; you will need to become your own CA. There is no other way around it. I tried searching disabling HSTS for localhost.dev, certificate for localhost.dev, but to no avail. Being your own CA, however, makes you HSTS proof (note that you can’t typically override an already set HSTS certificate, that is by design). However, in the old days you could simply mark your own self-signed certificate as trusted for your own domains. This is becoming less of an option these days. Becoming your own CA, however, still is an option.

Warning: The chain of trust

You should trust yourself not share your rootCA’s key and cert with anyone e…

Continue reading...

PostgreSQL CSV import

An article, posted almost 2 years ago filed in , , & .

Since I always forget (database management isn't my day-job): a short guide on how to quickly import large datasets in TXT or CSV into PostgreSQL. For smaller sets I still use ruby and FasterCSV to import the set, but nothing beats native DB imports in terms of speed. And speed doesn't matter when importing a few megabytes of data, but it certainly matters when it gets more than that.

In this example I'll use my current use-case, importing a large Drive-Time Matrix table, with drive times and distances between two postal codes. The head of the TXT file is formatted as such:

"Frompc4","Topc4","Time","Distance"
"1011","1011",0,0
"1011","1012",6,1737
"1011","1013",9,3378
"1011","1014",13,6056
"1011","1015",10,3198
"1011","1016",10,3112
"1011","1017",6,1706
"1011","1018",5,1791
"1011","1019",7,3146

Now let's assume we want to import this in a table 'DTM' with the following columns: from_pc (integer), to_pc (integer), time (in…

Continue reading...

Let’s encrypt! It’s easy!

An article, posted more than 2 years ago filed in , , , , , , , , & .

There important reasons to use HTTPS. It makes your systems more secure, helps to protect your users privacy, and will prevent others to hijack your account to deface your site.

If you’ve ever tried to secure your site you may have found how hard it is. You have to generate a private key, a certificate signing request, upload that request somewhere, pay, process the e-mail, upload the certificate, configure your server and set a reminder that in 1, 2, 3 or 5 years you’ve got to go through most of that same process again (which I described before in more detail in an earlier "how I do it"-article. Well, no longer! Enter: Let’s encrypt.

> Actually, Let’s encrypt is so easy that I had doubts whether I should even write this post. But maybe it wins an extra soul or two over.

The recommended way to get sta…

Continue reading...

Setting up https/spdy communication for your website with nginx

An article, posted more than 4 years ago filed in , , , , , , , & .

In case you do something with user accounts on your website, you definitely want to make sure you're using https. In general it protects the user's privacy, also when just reading content on your website. The only thing that can be seen by a middleman is that the person is viewing something at your server, the rest is all encrypted. And since Google has started to rank https-websites higher it has even become a SEO technique :) ). This article explains you how to serve your pages over https.

Update: a better option exists nowadays for non-domain validated certificates: Let's encrypt!

While the path to your server from someones desktop could be considered relatively ok in the past (harder to tap, putting a lot of trust in everything from the ISP to the internet exchanges and everything else in between), things have changed now. Wit…

Continue reading...

A somewhat secure Debian server with nginx, Passenger, rbenv for hosting Ruby on Rails with mail support and deployment with Capistrano

Basically this is a technical note to myself, in case I need to setup another server for running yet another personal Ruby on Rails project. And don't worry, I'm not going to replicate all nice guides out there, just filling in the gaps.

So let's start with the list of bookmarks I follow as a start. Note that in these tutorials mostly a user is used named 'deploy'. Typically I create a user per project and name databases etc. accordingly.

  1. Get security right first: My first 5 minutes on a server or essential security for Linux servers
  2. Then I get Rails up and running with this how to install Ruby on Rails with rbenv on Debian
  3. (in case you want to use the server as your remote git repo too) [Git setting up a remote repository and doing an inital push](http://the…

Continue reading...

How to do it: Using screen

An article, posted more than 5 years ago filed in , , , , , , , , & .

A technical note to myself: One way of doing multiple things simultanenously on a server can be by setting up multiple connections via SSH, that's how I used to do things before. An alternative is to use a single connection and use the command screen on the remote server. Another good reason to use screen is if you have a long running process that you don't want to break just because your SSH connection flips on and off with your computer going in and out of stand-by.

This is for absolute beginners. If you don't know about screen, this is for you. If you are already familiar with screen, I probably won't be able to educate you :o

So what is Screen?

GNU Screen is a kind of window managment system for the terminal (you're ought to say terminal multiplexer) and has several advantages over using multiple SSH connections. Most importantly: the processes keep running when SSH d…

Continue reading...

murb blog