I have always wanted to make this website load fast everywhere in the world, despite the server being in Montréal, Canada, without investing heavily. It shouldn’t be hard: after all, it is just a bunch of static files, generated with Jekyll.
Cloudflare brings a free CDN. You can set a page rule to aggressively cache your website on their CDN edge nodes, allowing your site to load as if it is hosted locally, even if you are half a world away.
There is just a little problem: how do you efficiently purge the cache when you update your site? It is quite easy to purge the entire cache on Cloudflare, but that is rather inefficient: most of your assets probably did not change, and now they will all have to be fetched again.
Today I decided to tackle this problem by creating
purge-static, a tool designed to purge your CDN cache. It can purge your Cloudflare cache for you. You can get started by running
pip install purge-static.
Recently, I had to deal with an issue on DMOJ, where a page displaying 100 out of around a million elements took over 10 seconds to load. Naturally, I started investigating the issue.
(Note: DMOJ uses MariaDB, but the same problem, as well as the eventual solution, should work the same on MySQL as well.)
The first course of action, of course, was to see what the database was trying to do, by running an
EXPLAINquery. For those of you who don’t know, if you have a query of the form
SELECT x FROM y WHERE z, running
EXPLAIN SELECT x FROM y WHERE zwould show what the query is doing, without actually executing the query.
A quick look at the
EXPLAINoutput showed that MariaDB first did a filter on a 2000 row table, and then joined in the table with a million elements. Then, the 100 wanted rows were filtered out. This query plan was quite horrifying.
On Windows, using Python 2’s subprocess module to launch a process with a unicode command line that is not strictly from the currently active ANSI code page (i.e. encoding
mbcs) will be mangled. All characters that cannot be encoded by
mbcswill, in fact, be replaced with
Obviously, this is can be resolved by switching to Python 3, but sometimes, converting to Python 3 is not yet an option. A terrifying prospect in 2018, but a problem nonetheless.
I present the module
uniprocess, which defines its custom version of
Popenand friends to work around the problem. I hope it proves useful to you.
I personally prefer the nginx.org packages for nginx over the ones that comes with Debian. They are usually newer and have a more sane amount of dependencies. I also prefer the
conf.dsystem over the
The main challenge in installing these packages on Debian is the trouble you have to go through to get the PGP keys and
sources.listset up. nginx.org does not present a good setup script. This has become a repetitive and annoying experience, so I present a series of commands to set it up quickly.
curl https://nginx.org/keys/nginx_signing.key | sudo apt-key add - (codename="$(dpkg --status tzdata | grep Provides | cut -f2 -d'-')"; echo; for deb in deb deb-src; do echo $deb http://nginx.org/packages/debian/ "$codename" nginx; done) | sudo tee -a /etc/apt/sources.list sudo apt update && sudo apt install nginx
curl https://nginx.org/keys/nginx_signing.key | sudo apt-key add - (codename="$(dpkg --status tzdata | grep Provides | cut -f2 -d'-')"; echo; for deb in deb deb-src; do echo $deb http://nginx.org/packages/mainline/debian/ "$codename" nginx; done) | sudo tee -a /etc/apt/sources.list sudo apt update && sudo apt install nginx
Since Python 2 is still in heavy use, at least in the deployment of the DMOJ judge, compatibility with it must be maintained. This necessitated writing code in such a fashion as to be compatible with both Python 2 and Python 3. The
sixlibrary has proved tremendously helpful in abstracting away the differences, some of which highly non-trivial.
six.with_metaclasshides away the difference in metaclass use. In Python 2, the
__metaclass__class member defines the metaclass used for the class, while in Python 3, one would specify it as
class Class(metaclass=MetaClass). The latter would be a syntax error in Python 2, and the former has no effect in Python 3.
sixprovides a solution that is highly non-obvious and yet works perfectly.
The most frustrating part is unicode-handling. The DMOJ judge was written somewhat sloppily in regards to unicode handling, dealing mostly with bytestrings and raw bytes. With the separation of
strin Python 3, strings in the judge must be turned into either
stron a case-by-case basis. It is decided that source code and program output will be treated as raw bytes, and textual data that are derived from these will be handled as UTF-8.
Most users are probably using Raspbian on their Raspberry Pi 3. However, Raspbian is designed for all Raspberry Pi devices, back to the original Raspberry Pi, which is ARMv6 with an FPU. This does not take advantage of the 64-bit support on the ARMv8 CPU on the Raspberry Pi 3.
Debian has offered ARM64 support for a while, and being the base distribution for Raspbian, is quite similar. Conveniently, there is a pre-built Debian image for Raspberry Pi 3. You can download it and copy it to a SD card, and it should work out of the box.
On Linux, the simple
ddcommand showed on the Debian Wiki works. On other platforms, notably Windows, Etcher is reputed to work well and has an easy interface.
The one flaw with this image is that the WiFi does not work.
Update: The 20180108 image now works with WiFi out of the box. The following instructions are no longer necessary.
ARM is unusual among the processors by having the program counter available as a “general purpose” register. Most other processors have the program counter hidden, and its value will only be disclosed as the return address when calling a function. If you want to modify it, a jumping instruction is used.
For example, on the x86, the program counter is called the instruction pointer, and is stored in
eip, which is not an accessible register. After a function call,
eipis pushed onto the stack, at which point it could be examined. Return is done through the
retinstruction which pops the return address off the stack, and jumps there.
Another example: on the MIPS, the program counter is stored into register
31after executing a
JALRinstruction, which is used for function calling. The value in there can be examined, and a return is a register jump
JRto that register.
ARM’s unusual design allows many, many ways of returning from functions. But first, we must understand how function calls work on the ARM.
Sometimes, after many attempts, you realized that to complete your mission, you must access
privatefields, or perhaps change
There are many reasons imaginable: the accessors copy the entire object before returning, and that takes a very long time, the authors forgot to provide an accessor, the library function is highly inefficient and you need to do better, …
Are you out of luck? Fortunately, no.
As most probably know, DMOJ uses a sandbox to protect itself from potentially malicious user submissions. An overview of the Linux sandbox has been published by my friend Tudor. However, it doesn’t go deep into the implementation details, many of which differ between Linux and FreeBSD.
At its core, the sandbox,
cptbox, uses the
ptrace(2)API to intercept system calls before and after they are executed, denying access and manipulating results. The core is written in C, hence the name
Perhaps the most obvious difference between Linux and FreeBSD is that on Linux,
ptrace(2)subfunctions are invoked as
ptrace(PTRACE_*), while on FreeBSD, it is
ptrace(PT_*). But this difference is rather superficial compared to the significant internal differences.
Most people, when first starting assembly, still carry over a lot of high level constructs in their assembly programs. A common pattern is to multiply and divide when a bit shift would suffice.
For example, a lot of people would write a program to write out the binary representation of an integer using the divide and modulo operations. This is rather inefficient compared to using shifts. For example, the divide by 2 can be replaced with a right shift by 1, and modulo 2 can be replaced by a bitwise AND with 1.
Aside: interestingly, taking any number modulo a power of two
mis equivalent to doing a bitwise AND with
m-1. The proof of this is left as an exercise for the reader.
This post will address the basics you need to know about shifts to get up to speed on writing good assembly.