Category: Uncategorized

I’ve compiled a small table for myself to check what’s best suited for me. Thought I’d share if someone tries to make the same decision.

	iCloud	GDrive	Dropbox	AWS Glacier
Possible to store in Cloud but not on local machine?	Yes (append .nosync to file or folder in iCloud-WebView)	Yes (pref -> google drive -> only sync these folders…)	Yes	Yes
Share Files by Link	Yes (“Add People”, works local and in webview)	Yes	Yes	No
Price for Storing 100 GB / month	3	2	10	0.5
Price for Storing 200 GB / month	3	10	10	1
Price for Storing 1 TB / month	10	10	10	5
Price for Storing 2 TB / month	10	20	20	10
Product Discontinuation Risk	Low	Low	Medium	Low
Tool for managing on Mac	iCloud Pref Pane, Context Menu, iCloud Drive	Backup and Sync App	Dropbox App	Freeze for Mac ($30) or CrossFTP
Configurable which folders to sync	Semi: iCloudDrive synced (if selected), also an option to sync “Desktop & Documents”	Yes (pref -> google drive -> only sync these folders…)	No (just the dropbox folder is synced)
Possible to sync files?	Yes	Yes	Yes	No
Retrieval costs / 100 GB	0	0	0	$10
Lock In	High	Medium	Medium	Huge (because retrieval is so expensive)
Need to switch when I switch to Linux?	Yes	No	No	No
Quality of the Web Interface	Medium	High	Don’t know	Low

iCloud

Most things with iCloud are fine and the functionality to backup your iPhone and sync photos when you’re a Mac user are of course cool. However, it’s not the best solution if you want to free disk space on your mac, since everything on iCloud is also synced on your Mac. You could use the ‘myfolder.nosync’ trick, but since the web-interface of iCloud isn’t that good it’s a bit problematic. For example, if you have a lot of photos, that you’d like to have in the cloud only and you have folders like 2015, 2016, 2017, 2018 on your local machine, each with sub folders of events. Then moving this to iCloud is a bit an awkward process, since you can’t just drag and drop folders to iCloud (it’ll say that’s not possible through the web interface). So you’ll have to move those directories locally to iCloud, and then add the nosync extension. If you had the photos on an external harddrive and they consume more disk space than you have on your machine, you’re a bit out of luck.

GDrive

I use Gdrive every day for my work, including Docs, Sheets, Drawings and everything. The web interface sometimes is a bit slow, but all in all solid. You can also upload directories, so it would be easy to move things from an external drive to GDrive by drag and drop.

Dropbox

I really don’t have much experience with Dropbox, so I’ll leave this part to someone else.

AWS Glacier

For the love of god, don’t use this as your personal storage solution. Downloading stuff is really slow (5h queue) and expensive, so not worth it unless you know what you’re doing!

Summary

For me as a Mac User a good setup is the following combo: get the 200 GB from iCloud to back up my Desktop, Documents, Photos and my Phone. To free disk space on my Mac, I’ll consider purchasing more space in GDrive, once I need it.

So you have a Continuous Integration Server that runs some build tasks. For example, you’re running ./gradlew buildTaskOne. Now, if you run ./gradlew buildTaskOne again you don’t want it to build again if nothing has changed, you just want it to say “up-to-date” and then move on to the next task. On your local machine this goal is achieved relatively straightforward. For Java builds this works out of the box, for other builds a small setup like the following is required:

task installNodeModules(type: Exec) {
    commandLine 'npm', 'i'
    inputs.files 'package.json'
    outputs.dir 'node_modules'
}

The fist time this task is ran this is executed, the second time it is skipped:

That was easy! So how can you achieve the same thing when using gitlab runners in combination with AWS Autoscale? Some more steps are required to achieve this…

Set Up Distributed S3 Caches

AWS Autoscale uses distinct EC2 instances for each build stage that are dynamically spun up and torn down. This means that for sharing caches, we need a central file store. And what could be better suited for this task than S3? Also, Gitlab integrates really well with this. Simply put the following in your config.toml of your AWS Autoscale manager (bastion server):

concurrent = 10
check_interval = 0
log_level = "debug"

[[runners]]
  name = "gitlab-aws-autoscaler"
  url = "<your url>"
  token = "${RUNNER_TOKEN}"
  executor = "docker+machine"
  limit = 6
  [runners.docker]
    image = "taskbase/gitlabrunner:2.4"
    privileged = true
    disable_cache = true
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache", "/builds:/builds"]
  [runners.cache]
    Type = "s3"
    ServerAddress = "s3.amazonaws.com"
    AccessKey = "${AWS_ACCESS_KEY_ID}"
    SecretKey = "${AWS_SECRET_ACCESS_KEY}"
    BucketName = "your-bucket-name"
    BucketLocation = "eu-central-1"
    Shared = true

The bold part is interesting for you, I’ve added the rest to give you a bit more context. The red bits need to be adjusted to your setup. You’ll need to create an S3 bucket for this to work first.

Set Up Caches in .gitlab-ci.yml

You’ll also need to specify what you want to cache in .gitlab-ci.yml. This is minimal setup for the node_modules example:

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
  - .gradle
  - ./node_modules # important: don't forget to cache the output!

Those directories will get uploaded to S3. Now I’m not entirely sure about this, but I think the directories have to be within your repository. For example I tried to specify an absolute path /opt/gradle/caches but this didn’t get uploaded! In order to store gradle caches into the .gradle folder, run the following line in a before script part of gitlab ci:

before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle # there is no $USER_HOME, and /opt/gradle/caches can't be uploaded, so local gradle home is chosen

Explanation for the above setup (optional reading)

On AWS Autoscale runners you don’t have a $USER_HOME variable and gradle seems to store caches by default into /opt/gradle/caches. You can verify this with the following gradle task:

task showMeCache << {
    configurations.compile.each { println it }
}

But when trying to store the cache at /opt/gradle/caches nothing seems to happen, so a local gradle home is chosen. This can be achieved by setting the GRADLE_USER_HOME variable to a location within your repo. That’s what the command in the before script is good for.

It’s also important to cache the entire .gradle folder and not just ./gradle/caches/ since to check whether something is up-to-date gradle checks the file hashes in ./gradle/4.6/fileHashes, at least in gradle version 4.6. I learned this the hard way by running a git diff on the .gradle folder before and after making a change to the file declared in input.files.

Setting Up the build.gradle File

Another important concept when setting up the gradle caching mechanisms is “path sensitivity”. By default, the path sensitivity is ABSOLUTE and not RELATIVE, meaning that when you have a cache at /my/path/bla and then /your/path/bla the cache will NOT take effect. From what I’ve seen the builds on AWS Autoscale for the same branches have the same absolute path so it’s not a problem, but to be sure you could write something like the following:

@PathSensitive(PathSensitivity.RELATIVE)
@InputFiles
FileTree packageJSON() {
    return fileTree('./package.json')
}
task npmInstall (type: Exec) {
    inputs.files packageJSON()
    outputs.dir 'node_modules'
    commandLine 'npm', 'i'
}

or

task npmInstall (type: Exec) {
    inputs.files 'package.json' withPathSensitivity(PathSensitivity.RELATIVE)
    outputs.dir 'node_modules'
    commandLine 'npm', 'i'
}

Of course, only apply this if your input files are really path independent. But you can also leave this away and introduce it once necessary.

Checking if Everything is Working

For some reason, the cache doesn’t seem to kick in the first time it should. So when we push changes, and then on the second run it should be cached, the caches somehow don’t work yet. But on the third run, they do work.

Summary

What is minimally required to successfully employ input / output caches are the following steps:

• S3 distributed cache setup correctly
• Gradle Home within repository, tried and tested with .gradle
• Upload gradle home folder to S3
• Upload outputs of build task to S3 as well. The build task needs an output in order be be cachable.
• Try at least two times to check if caching works, the first time when the caches already should kick in, they don’t work for some reason…

StackOverflow sorting sucks.

1. The accepted answer is always on top. This is a joke. The asker of the question usually accepts the first best thing and then heads off. Thousands of people to come will then see the first best answer on top. For example: https://stackoverflow.com/questions/16267339/s3-static-website-hosting-route-all-paths-to-index-html . How ridiculous is that?
2. Below the accepted answer, it’s sorted by upvotes. That system is also broken, because new answers will have a very hard time of getting more upvotes.

The solution? A smart sorting algorithm, like on Quora or Reddit.

There’s an ongoing discussion on StackOverflow about this:

https://meta.stackoverflow.com/questions/326095/please-unpin-the-accepted-answer-from-the-top

https://meta.stackoverflow.com/questions/283456/order-highest-upvoted-answer-before-accepted-answer

So, StackOverflow, are you ever going to change this?

I was excited when I heard Angular 2 would support server-side rendering since SEO was a huge PITA (pain in the ass) with Angular 1. It was said that server-side rendering would be possible through Angular Universal.

Yet, this Angular Universal thing, what’s up with that? Seems to have a really small team on that and there doesn’t really seem to be any documentation. Well at least there’s a getting started repo, so I thought. However, this repo (https://github.com/robwormald/ng-universal-demo) doesn’t even have a readme…

Also, when you check out some of those files, they don’t seem very professional. For example: https://github.com/robwormald/ng-universal-demo/blob/master/src/express-engine.ts

First line of code: //hacky express wrapper thingy.

Seriously?

Then of course, there’s also some confusion as to where to go and look for resources. You’ll find quite a few.

When you just google “Angular Universal” (as of September 2017) you’ll get to https://universal.angular.io/. However, this page is a bit out of date.

Just have a look at the date:

Or the actual site:

Angular 2 is now Angular 4 and --ambient is also not a thing anymore. So the only purpose of universal.angular.io is to confuse people with outdated material and to link to github (https://github.com/angular/universal).

Github, even though being the best resource on Angular Universal, isn’t really documented either. Meh.

Well, I guess switching to React isn’t such a bad idea after all?

Why do Java programmers wear glasses?

Because they can’t C#.

SQL

A SQL query goes into a bar, walks up to two tables and asks, “Can I join you?”

3 guys

A physicist, an engineer and a programmer are on a steep mountain passage, when suddenly the breaks stop functioning. After some shouting and near death experiences, they finally manage to bring the car to a halt. The physicist says “Let’s check the temperature on the breaks and test the friction, maybe then we can figure out what went wrong.”. The engineer goes “Let me just get some straps, I can fix it”. The programmer just gently shakes his head. Then he goes “Guys that’s not how it works. Let’s first do it again and check if it’s reproducible!”.

OOP Joke

Q: What’s the object oriented way to become rich?
A: Inheritance

[“hip”, “hip”]

(hip hip, array!)

The following code converts a base64 String to a long and vice versa.

private static final String alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-";
public static String encodeBase64(long v) {
    char[] a = alphabet.toCharArray();
    v = Math.abs(v);
    String s = "";
    for (int i = 0; i < 11; i++) {
        long val = v & 63;
        s = a[(int) val] + s;
        v >>= 6;
    }
    while (s.startsWith("A") && s.length() > 1)
        s = s.substring(1, s.length());
    return s;
}
public static long decodeBase64(String s) {
    char[] a = alphabet.toCharArray();
    Map<Character, Integer> map = new HashMap<>();
    for (int i = 0; i < a.length; i++)
        map.put(a[i], i);
    char[] chars = s.toCharArray();
    long v = 0;
    for (char c : chars) {
        v <<= 6;
        v = v | map.get(c);
    }
    return v;
}