In this series I explain how I do certain, mostly technical, stuff. Basically it is written for the audience that is just like me (in other words: this is just my public external memory ;) ). For those, who like me, forgets about the details when doing this somewhat more advanced server stuff, but is capable of throwing in some commands at the command line and cares about doing stuff in the right way.
Enjoyed this? Follow me on Mastodon or add the RSS, euh ATOM feed to your feed reader.
I don't mind running my own virtual servers. Fail2ban is a tool I've had running on my servers for years. It helps fencing of requests from ip-addresses that repeatedly misbehave when connecting to SSH and postfix. I never got to creating my own rules. I thought I had to write it in some arcane scripting language, but recently I learned it is pretty easy.
In this case I wanted to block 500 (internal server error) and 422 (Unprocessable Entity) errors. A server error once in a while is expected, but repeated server errors are suspicious. Common source of these errors are scripts that scan for things like SQL injections.
Examples given are for Debian.
/etc/fail2ban/filter.d/nginx-errors.conf
[Definition]
failregex = ^ -.*"(GET|POST|HEAD).*HTTP.*" (500|422)
port = http,https
ignoreregex =
backend = auto
logpath = /var/log/nginx/access.log
bantime = 600
maxretry = 10
And appending to /etc/fail2ban/jail.local…
Just a short article to document for myself how to copy a large directory (e.g. a user-folder) over a local network. While (s)cp might work for smaller operations, rsync is my preferred tool as you can restart it when it breaks + in case you found an optimization, you can just abort and restart. Some things to take into account before I share the command:
So here is the command:
rsync -aWP --inplace --exclude-from=exclude-file.txt murb@someaddress:/Users/username/ .
Breakdown:
-a is the archival option, and it is typically what y…Every now and then I’m searching for this little snippet in my notes using NotationalVelocity (or currently actually a fork):
du -hsx * | sort -rh | head -100
It’s a variation of a snippet I found somewhere, but hardly invested any time in understanding what it actually does. Let’s decompose, from head to taildu.
headhead -100
head simply limits the results to a maximum of 100 lines. Not much more to explain here
sortsort sorts. by default it sorts the files by filename, but adding ‘-h’ to it allows it to sort by “human readable numbers” (e.g. 5M > 6K); if ‘-n’ would be added as option 6K would be > 5M. The ‘-r’ options reverses the sort wich is by default ascending.
dudu by defaults crawls a directory recursively for all files. passing '-s' tells it to sum the values of files within directories. the '-x' option is used to n…
It has been quite some time ago, but here is another 'how i do it' article :)
If, by 'accident' you have, like me, chosen projectname.dev for your local development as a convention, and you want to continue using this convention; you will need to become your own CA. There is no other way around it. I tried searching disabling HSTS for localhost.dev, certificate for localhost.dev, but to no avail. Being your own CA, however, makes you HSTS proof (note that you can’t typically override an already set HSTS certificate, that is by design). However, in the old days you could simply mark your own self-signed certificate as trusted for your own domains. This is becoming less of an option these days. Becoming your own CA, however, still is an option.
You should trust yourself not share your rootCA’s key and cert with anyone e…
Since I always forget (database management isn't my day-job): a short guide on how to quickly import large datasets in TXT or CSV into PostgreSQL. For smaller sets I still use ruby and FasterCSV to import the set, but nothing beats native DB imports in terms of speed. And speed doesn't matter when importing a few megabytes of data, but it certainly matters when it gets more than that.
In this example I'll use my current use-case, importing a large Drive-Time Matrix table, with drive times and distances between two postal codes. The head of the TXT file is formatted as such:
"Frompc4","Topc4","Time","Distance"
"1011","1011",0,0
"1011","1012",6,1737
"1011","1013",9,3378
"1011","1014",13,6056
"1011","1015",10,3198
"1011","1016",10,3112
"1011","1017",6,1706
"1011","1018",5,1791
"1011","1019",7,3146
Now let's assume we want to import this in a table 'DTM' with the following columns: from_pc (integer), to_pc (integer), time (in…
There important reasons to use HTTPS. It makes your systems more secure, helps to protect your users privacy, and will prevent others to hijack your account to deface your site.
If you’ve ever tried to secure your site you may have found how hard it is. You have to generate a private key, a certificate signing request, upload that request somewhere, pay, process the e-mail, upload the certificate, configure your server and set a reminder that in 1, 2, 3 or 5 years you’ve got to go through most of that same process again (which I described before in more detail in an earlier "how I do it"-article. Well, no longer! Enter: Let’s encrypt.
> Actually, Let’s encrypt is so easy that I had doubts whether I should even write this post. But maybe it wins an extra soul or two over.
The recommended way to get sta…
In case you do something with user accounts on your website, you definitely want to make sure you're using https. In general it protects the user's privacy, also when just reading content on your website. The only thing that can be seen by a middleman is that the person is viewing something at your server, the rest is all encrypted. And since Google has started to rank https-websites higher it has even become a SEO technique :) ). This article explains you how to serve your pages over https.
Update: a better option exists nowadays for non-domain validated certificates: Let's encrypt!
While the path to your server from someones desktop could be considered relatively ok in the past (harder to tap, putting a lot of trust in everything from the ISP to the internet exchanges and everything else in between), things have changed now. Wit…
Basically this is a technical note to myself, in case I need to setup another server for running yet another personal Ruby on Rails project. And don't worry, I'm not going to replicate all nice guides out there, just filling in the gaps.
So let's start with the list of bookmarks I follow as a start. Note that in these tutorials mostly a user is used named 'deploy'. Typically I create a user per project and name databases etc. accordingly.
A technical note to myself: One way of doing multiple things simultanenously on a server can be by setting up multiple connections via SSH, that's how I used to do things before. An alternative is to use a single connection and use the command screen on the remote server. Another good reason to use screen is if you have a long running process that you don't want to break just because your SSH connection flips on and off with your computer going in and out of stand-by.
This is for absolute beginners. If you don't know about screen, this is for you. If you are already familiar with screen, I probably won't be able to educate you :o
GNU Screen is a kind of window managment system for the terminal (you're ought to say terminal multiplexer) and has several advantages over using multiple SSH connections. Most importantly: the processes keep running when SSH d…
Dit artikel van murblog van Maarten Brouwers (murb) is in licentie gegeven volgens een Creative Commons Naamsvermelding 3.0 Nederland licentie .