Pi(e) Holing Facebook

It all started with a click. While reading the newspaper i clicked on a link to Facebook and was shocked when it opened.

The reason for my surprise was that in my /etc/hosts i had the following entry:

# Block Facebook
127.0.0.1   www.facebook.com
127.0.0.1   facebook.com

a rather blunt instrument, but one that until now had been effective at shitcanning any links. So why had it stopped working? After some confused poking around it became obvious that my new ISP provided way more IPv6 routing than the old ISP, and macOS was now favouring IPv6 traffic. As a consequence the hack in my /etc/hosts grew to include entries for IPv6:

fe80::1%lo0 www.facebook.com
fe80::1%lo0 facebook.com

And once more Facebook was back in the shitcan.

Note: adding hosts to /etc/hosts is obviously tedious – you can’t wildcard and blocking the root domain doesn’t block sub-domains. In order to get rid of all Facebook servers (just the obvious ones) takes over ten entries, all of which need to now be repeated for IPv6.

At this point any rational person would conclude that this is not a sane thing to be doing. Obviously it’s time to be running my own DNS server and sinkhole and shitcanning domains with wildcards!

Fortunately there are still plenty of people on the internet who haven’t given up, for example, Pi-hole. By installing Pi-hole on a Raspberry PI hanging off the back of my router, and updating clients to use it as a DNS, i have a place where it is possible to wildcard block entire domains.

As a well as providing DNS Pi-hole also maintains a (partial) list of domains that serve ads. This means that devices on your home network that aren’t running ad blocking now has a good chance of not being served ads. This was a partially solved problem, as the Raspberry PI also runs Privoxy  which also blocks a good percentage of ads.

As an aside, the war between ad blockers and ad pushers has been quietly escalating and i’ve been starting to notice that a few news sites are managing to execute Javascript that blocks uBlock Origin. Sites that employ such measures are still blocked from displaying ads by Pi-hole and / or Privoxy.

While installing Pi-hole it was necessary to make some decisions about what to use as a DNS authority. There are some obvious answers like 8.8.8.8 (Google), 9.9.9.9 (IBM and some shady law enforcement types), OpenDNS, OpenNIC, etc. None of which seem ideal.

You probably won’t be surprised to hear that all your DNS queries are sent, unencrypted, over port 53. Which initially sounds like a really bad thing – it would provide your ISP with an easy way to know every site that you looked up. However, in all likelihood they aren’t doing that… mostly because they have stronger, government mandated, requirements to meet, such as tracking every site that you actually visit and when you visited it, not just the ones that you happen to lookup, and then subsequently visit via a cached lookup. If all you had to do was run your own DNS to avoid tracking… yeah, not going to happen.

Despite the above rational, there exists a parallel DNS infrastructure called DNSCrypt, mostly volunteer run, that proxies encrypted access to DNS. Assuming that you can trust that they aren’t logging (something you’re already doing with the DNS providers listed above…) then you can effectively block any visibility of your DNS activity to your ISP… not that they’ll care. If your traffic isn’t leaving your machine via an encrypted tunnel (think VPN, Tor, etc) then you can assume that it is being inspected and logged at the packet level.

In terms of increasing privacy DNSCrypt doesn’t seem to offer very much. It does offer some other protections against DNS spoofing attacks, but i’m not sure how widespread those are in the wild. I’d also guess that the other major providers of DNS are taking countermeasures as they are needed… and are maybe more effective than the volunteer force behind DNSCrypt.

I’ll probably end up installing the dnscrypt-proxy on the Raspberry PI and using it as the resolver for Pi-hole. In the end it’s just going to be an encrypted proxy for OpenNIC, which if given a choice is where i’d want my DNS to be resolved.

I’d recommend looking into Pi-hole it’s a really nice of tools to have a better understanding and control of what devices on your network are actually doing. Oh, and keep in mind that IPv6 is now a thing, running in parallel to the IPv4 internet for which you probably had some reasonable mental model… learning about RA, SLAAC and it’s Privacy Extensions) DAD, etc. was an eye opener for me!

Advertisements

Youtube… ffs

For the longest time i’ve been using a Safari Extension called ClickToPlugin, which replaced Youtube’s video player with the native Safari video player. There were a couple of reason for this, the biggest of which was the horrendous amount of CPU that the YouTube HTML5 player uses. It also disabled autoplay, another scourge of the ad-supported web. Oh, and it never played ad.

The recent re-design broke all this, and it doesn’t look like it’ll be repaired. Time to find another solution… <sigh>

There are other Youtube focused extension out there for Safari, but none of them seem to exactly what i want. Firefox has a few plugins to allow downloading, or copying the video URL, which gives you a way to choose the player. There doesn’t, however, seem to be anything that does exactly what ClickToPlugin managed.

For a few weeks i’ve been using a Firefox plugin to copy the video URL, pasting that into Safari, and letting it play it with the native player. But it means opening Firefox, and switching between browsers, etc.

More recently i started playing with youtube-dl. If i’m going to copy and pasting URLs why not give them to a script, and have it spawn a Quicktime player? Well, the Quicktime player doesn’t have a command line… and who wants to wait until a video has downloaded before watching? It would be better to pipe the output of youtube-dl to a player… but that will have to be something other than Quicktime.

When in doubt try ffmpeg – the true swiss army knife of video! The ffmpeg distribution includes a tool ffplay, which can play video piped into stdin. Looks like we have everything needed:

$ youtube-dl -q -f best --buffer-size 16K https://www.youtube.com/watch?v=DLzxrzFCyOs -o - | ffplay -fast -fs -loglevel quiet -

Now all i need is a dumb bash script in my path, which takes a URL, and plugs it into that command:

#!/bin/bash
if [ $# -ne 1 ]; then
    echo Usage: yt url
    exit 1
fi

url=$1

youtube-dl -q -f best --buffer-size 16K $url -o - | \
 ffplay -fs -loglevel quiet -

Yes, the amount of time and effort involved in avoiding the unavoidable smartness of the smartest people in Silicon Valley…

Signal Desktop (Again)

It could be that i’m still confused about the release channel for the standalone Signal-Desktop client, but it doesn’t appear to be released as a bundle.

My previous build from the ‘electron’ branch stopped working, telling me i needed to update. However, the electron branch has gone… which is actually good news, as it means that the changes have merged to master.

Starting from scratch, but with all of the NodeJS NPM / Yarn junk still around, all it took was cloning from GitHub:

$ git clone –recursive https://github.com/WhisperSystems/Signal-Desktop.git

$ cd Signal-Desktop/
$ yarn pack-prod

Edit: the module dance can be avoided with the following:

$ node –version
v8.6.0
$ yarn –version
1.1.0
$ git clone –recursive https://github.com/WhisperSystems/Signal-Desktop.git

$ cd Signal-Desktop/
$ yarn install
$ yarn pack-prod

And then the usual dance to add modules until things started working. That part of the process defies description, and short of trying re-install all the npm / yarn stuff and starting from nothing, its unlikely that we’ll see the same things. In my case i had to ‘npm install’ the following:

  • electron-icon-maker
  • grunt

and ‘yarn add’:

  • grunt-sass

I’d have thought that there is a actually a module dependency list in the bundle, and a yarn / npm command to process it… no idea what that might be!

signal-desktop

It would be nice if there was an official build soon. Would like to stop having to do this dance – especially as the builds have been working perfectly for what i need for months now!

… No!

Last weekend / week some valiant efforts were made to import a Blogger export file into the latest WordPress release running on my laptop. Just getting that shitshow on the road was not a walk in the park. I’ll leave the next paragraph as a warning to future travellers on macOS:

WordPress will not connect to MySQL / MariaDB on localhost, update your config to use 127.0.0.1.

How PHP manages to be so consistently bad is a source of amazement.

My initial excitement about having found a plugin was tempered by the eventual realisation that the hooks are only there when saving a new entry. Presumably the code could be re-used in a pipeline that uses the Blogger REST API to pull entries, pushing them into WordPress, attaching images and rewriting img tags as it goes.

My hopes of getting this done without writing my own toolchain are slowly fading into the ether…

Just

You do it to yourself, you do
And that’s what really hurts
Is you do it to yourself, just you
You and no-one else
You do it to yourself
You do it to yourself

That’s the thing, if i was better at ignoring things…

Last weekend’s adventure was making a local backup of all my images in Flickr. This was really about getting to a point where it wouldn’t matter if that account got closed, or the new owners (Verizon?) decided to move everything, etc. It was a good start, but there is another (bigger) problem that is going to stop me from moving forward: the IWTFY blog.

There are hundred… i don’t know, maybe thousands of posts there that link directly to the images hosted on Flickr. Oops. Besides that being against the terms of service it rather neatly ties me to keeping the Flickr account for as long as there is an IWTFY blog… and that’s ongoing for a decade at this point!

Obviously a plan is needed!

  1. Export IWTFY from Blogger / Google – this seems straightforward.
  2. Setup a local instance of WordPress where i can run plugins.
  3. Write a plugin that pulls all of the images into the local WordPress instance
  4. Export IWTFY from the local instance
  5. Import into a hosted WordPress

This feels very much like a process that someone else must have already been through, and either made available / sells as a service… right? Proceeding onwards assuming that i’m not able to find that person / service, it looks like it isn’t difficult to use something like  the WP_Http API call to grab images from within a blog post. And somewhere in here there has to be a way loop through all the entries in blog. No idea what kind of WordPress *thing* i’m supposed to be coding, but yeah… Bildung!

Can’t actually believe i’m going to do this. Save me from myself? Please.

[Yes, the tags are something of a honeypot.]

Edit: someone already wrote the plugin i need! Import External Images – even if it doesn’t do exactly what is needed, its only a little PHP hacking… how bad could it be?!

Signal Desktop (Update)

My weekend adventures got me as far as a running Election branch. Something was obviously a little weird as it was opening with a javascript debugger taking up the right third of the app window. Not ideal, but easy to close.

And then i tried to link it to the account on my phone… no dice.

What i’d built was the development version, which defaults to connecting to the test / staging servers. For any of this experiment to be useful it needs to connect to the production servers.

These are all defined in the /config directory, but it wasn’t obvious to me how to change to another config. In the package.json there were references to ${env.SIGNAL_ENV} but exporting that to the shell with a value of ‘production’ didn’t help.

A little reading about my new friend ‘yarn’ showed that there was a ‘–production=true’ CLI option. Passing that along with ‘yarn install’ broke things.. but the command:

$ yarn start --production=true

Moves things along and makes it possible to link to a phone account. The process is actually pretty smooth – you just take a picture of a QRCode, give the client a name and you’re done.

What i’ve still not worked out is how to tell my other new friend ‘electron-packager’ to package the production version. Until that is worked out i’m stuck with the Muon version… which isn’t a bad place – it has been very stable thus far.

Edit: It seems ‘electron-packager’ was not my friend after all, ‘yarn’ however i’m starting to like! The ‘package.json’ contains a ‘scripts’ section defining a bunch of targets, one of which ‘pack-prod’ is ‘SIGNAL_ENV=production npm run dist’ which seems exactly what i need. And, indeed:

$ yarn pack-prod

spat out a ‘dist’ dir containing a working ‘production’ Signal.app.

The end.

Flickr Local Backup?

Flickr is dead / dying. Yahoo! is owned by someone like CBS… i’m not even sure i want to know. Time to get out. Maybe not completely, but that’s a different story.

My plan is to make a local backup / archive of everything that i’ve uploaded. In theory there is a copy of everything elsewhere, but as previous discussed, my organisational skills leave a little to be desired.

Pick a tool. There is a Python Flickr API kit thing, so presumably someone will have written a backup tool. Checkout GitHub and see what is there. The most recently updated / highly rated choice is Flickrmirrorer. Lets give that a go:

$ git clone --recursive https://github.com/markdoliner/flickrmirrorer.git
Cloning into 'flickrmirrorer'...
remote: Counting objects: 483, done.
remote: Total 483 (delta 0), reused 0 (delta 0), pack-reused 483
Receiving objects: 100% (483/483), 104.01 KiB | 0 bytes/s, done.
Resolving deltas: 100% (225/225), done.
$ cd flickrmirrorer/
$ ./flickrmirrorer.py --help
Traceback (most recent call last):
 File "./flickrmirrorer.py", line 49, in <module>
 import requests
ImportError: No module named requests

Ah, fun… Python libraries. This always ends well. According to the README.md we need:

  • python 2.something or python 3.anything
  • python dateutil
    • Ubuntu: apt-get install python-dateutil
  • python flickrapi library 2.0 or newer.
  • python requests

I’ve not done any Python dev on this machine, and it’s macOS not Linux, so no apt-get here. The first thing to try is installing pip… with easy_install. Sometimes it’s pip, sometimes it’s easy_install. They might well be equivalent… no idea!

$ sudo easy_install pip
...
$ sudo pip install requests
...
$ sudo pip install dateutils
...
$ sudo easy_install flickrapi

Might have just as easily been able to install flickrapi with pip, but they said they wanted easy_install. And now:

$ ./flickrmirrorer.py --help
usage: flickrmirrorer.py [-h] [-v] [-q] [-s] [--ignore-views]
 [--ignore-photos] [--ignore-videos]
 [--delete-unknown]
 destdir

Create a local mirror of your flickr data.

positional arguments:
 destdir the path to where the mirror shall be stored

optional arguments:
 -h, --help show this help message and exit
 -v, --verbose print progress information to stdout
 -q, --quiet print nothing to stdout if the mirror succeeds
 -s, --statistics print transfer-statistics at the end
 --ignore-views do not include views-counter in metadata
 --ignore-photos do not mirror photos
 --ignore-videos do not mirror videos
 --delete-unknown delete unrecognized files in the destination directory.
 Warning: if you choose to ignore photos or videos, they
 will be deleted!

Cool. Now all we need to do is setup a disk to keep all this stuff and kick it off:

$ ./flickrmirrorer.py /Volumes/FlickrDump/backup/
0:121: execution error: "https://www.flickr.com/services/oauth/authorize?oauth_token=<token>&perms=read" doesn’t understand the “open location” message. (-1708)

Please authorize Flickr Mirrorer to read your photos, titles, tags, etc.

1. Visit https://www.flickr.com/services/oauth/authorize?oauth_token=<token>&perms=read
2. Click "OK, I'LL AUTHORIZE IT"
3. Copy and paste the code here and press 'return'

XYZ-ABC-123

Fetching 33991776973.jpg
Updated metadata for 33991776973.jpg
Fetching 32923862986.jpg
Updated metadata for 32923862986.jpg
Fetching 31996362424.jpg
...

This will presumably take a while. An hour or so in:

$ ls -1 *.jpg | wc -l
 900

Edit: of course it died right after this was posted…

Traceback (most recent call last):
 File "./flickrmirrorer.py", line 812, in <module>
 main()
 File "./flickrmirrorer.py", line 807, in main
 mirrorer.run()
 File "./flickrmirrorer.py", line 195, in run
 self._run_helper()
 File "./flickrmirrorer.py", line 237, in _run_helper
 self._download_all_photos()
 File "./flickrmirrorer.py", line 288, in _download_all_photos
 new_files |= self._download_photo(photo)
 File "./flickrmirrorer.py", line 398, in _download_photo
 photo_datetime = get_photo_datetime(photo)
 File "./flickrmirrorer.py", line 147, in get_photo_datetime
 return dateutil.parser.parse(photo['datetaken'])
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py", line 697, in parse
 return DEFAULTPARSER.parse(timestr, **kwargs)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py", line 310, in parse
 ret = default.replace(**repl)
ValueError: month must be in 1..12

<sigh>

It seems that some of the images in Flickr has no date taken data at all. Isn’t that fun. Time for a little hackery:

$ git diff flickrmirrorer.py 
diff --git a/flickrmirrorer.py b/flickrmirrorer.py
index ef9aa85..29df9a2 100755
--- a/flickrmirrorer.py
+++ b/flickrmirrorer.py
@@ -144,7 +144,11 @@ def get_photo_datetime(photo):
 datetime.datetime
 """
 if photo['datetakenunknown'] == "0":
- return dateutil.parser.parse(photo['datetaken'])
+ try:
+ return dateutil.parser.parse(photo['datetaken'])
+ except ValueError:
+ # now what?
+ return datetime.datetime(1970,1,1,0,0,0,0)
 
 try:
 parsed = datetime.datetime.strptime(photo['title'], '%Y%m%d_%H%M%S')

Which is to say:

  • wrap the parsing of ‘datetaken’ in a try / catch
  • return the beginning of unix time and move on

Suppose it might have been easier to edit the metadata on flickr… but where is the fun in that!