End-of-central-directory signature not found

You download a zip file only to find that it barfs when you try to un-archive it:

$ unzip -t test.zip 
Archive:  test.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.

What a nightmare.

 

Now you’re left with the prospect of attempting to download again, or see if you can salvage what you have. Bandwidth is cheap, but the machine at the other end is no longer responding. Great… time to learn how to extract a partially downloaded / corrupt zip file!

It’s actually a lot easier than you might think… which makes me wonder why i’ve never learnt it before. First try a little force:

$ zip -F test.zip --out partial.zip
Fix archive (-F) - assume mostly intact archive
	zip warning: bad archive - missing end signature
	zip warning: (If downloaded, was binary mode used?  If not, the
	zip warning:  archive may be scrambled and not recoverable)
	zip warning: Can't use -F to fix (try -FF)

zip error: Zip file structure invalid (test.zip)

Nope. Now a little more force:

$ zip -FF test.zip --out partial.zip
Fix archive (-FF) - salvage what can
zip warning: Missing end (EOCDR) signature - either this archive
is not readable or the end is damaged
Is this a single-disk archive? (y/n): y
Assuming single-disk archive
Scanning for entries...
copying: selected exported/3 monkeys.jpg (2629234 bytes)
...
copying: selected exported/worried and walking.jpg (21563355 bytes)
Central Directory found...
zip warning: reading central directory: Undefined error: 0
zip warning: bad archive - error reading central directory
zip warning: skipping this entry...

Good to go?

$ unzip -qt partial.zip 
No errors detected in compressed data of partial.zip.

Good to go!

Advertisements

Pi(e) Holing Facebook

It all started with a click. While reading the newspaper i clicked on a link to Facebook and was shocked when it opened.

The reason for my surprise was that in my /etc/hosts i had the following entry:

# Block Facebook
127.0.0.1   www.facebook.com
127.0.0.1   facebook.com

a rather blunt instrument, but one that until now had been effective at shitcanning any links. So why had it stopped working? After some confused poking around it became obvious that my new ISP provided way more IPv6 routing than the old ISP, and macOS was now favouring IPv6 traffic. As a consequence the hack in my /etc/hosts grew to include entries for IPv6:

fe80::1%lo0 www.facebook.com
fe80::1%lo0 facebook.com

And once more Facebook was back in the shitcan.

Note: adding hosts to /etc/hosts is obviously tedious – you can’t wildcard and blocking the root domain doesn’t block sub-domains. In order to get rid of all Facebook servers (just the obvious ones) takes over ten entries, all of which need to now be repeated for IPv6.

At this point any rational person would conclude that this is not a sane thing to be doing. Obviously it’s time to be running my own DNS server and sinkhole and shitcanning domains with wildcards!

Fortunately there are still plenty of people on the internet who haven’t given up, for example, Pi-hole. By installing Pi-hole on a Raspberry PI hanging off the back of my router, and updating clients to use it as a DNS, i have a place where it is possible to wildcard block entire domains.

As a well as providing DNS Pi-hole also maintains a (partial) list of domains that serve ads. This means that devices on your home network that aren’t running ad blocking now has a good chance of not being served ads. This was a partially solved problem, as the Raspberry PI also runs Privoxy  which also blocks a good percentage of ads.

As an aside, the war between ad blockers and ad pushers has been quietly escalating and i’ve been starting to notice that a few news sites are managing to execute Javascript that blocks uBlock Origin. Sites that employ such measures are still blocked from displaying ads by Pi-hole and / or Privoxy.

While installing Pi-hole it was necessary to make some decisions about what to use as a DNS authority. There are some obvious answers like 8.8.8.8 (Google), 9.9.9.9 (IBM and some shady law enforcement types), OpenDNS, OpenNIC, etc. None of which seem ideal.

You probably won’t be surprised to hear that all your DNS queries are sent, unencrypted, over port 53. Which initially sounds like a really bad thing – it would provide your ISP with an easy way to know every site that you looked up. However, in all likelihood they aren’t doing that… mostly because they have stronger, government mandated, requirements to meet, such as tracking every site that you actually visit and when you visited it, not just the ones that you happen to lookup, and then subsequently visit via a cached lookup. If all you had to do was run your own DNS to avoid tracking… yeah, not going to happen.

Despite the above rational, there exists a parallel DNS infrastructure called DNSCrypt, mostly volunteer run, that proxies encrypted access to DNS. Assuming that you can trust that they aren’t logging (something you’re already doing with the DNS providers listed above…) then you can effectively block any visibility of your DNS activity to your ISP… not that they’ll care. If your traffic isn’t leaving your machine via an encrypted tunnel (think VPN, Tor, etc) then you can assume that it is being inspected and logged at the packet level.

In terms of increasing privacy DNSCrypt doesn’t seem to offer very much. It does offer some other protections against DNS spoofing attacks, but i’m not sure how widespread those are in the wild. I’d also guess that the other major providers of DNS are taking countermeasures as they are needed… and are maybe more effective than the volunteer force behind DNSCrypt.

I’ll probably end up installing the dnscrypt-proxy on the Raspberry PI and using it as the resolver for Pi-hole. In the end it’s just going to be an encrypted proxy for OpenNIC, which if given a choice is where i’d want my DNS to be resolved.

I’d recommend looking into Pi-hole it’s a really nice of tools to have a better understanding and control of what devices on your network are actually doing. Oh, and keep in mind that IPv6 is now a thing, running in parallel to the IPv4 internet for which you probably had some reasonable mental model… learning about RA, SLAAC and it’s Privacy Extensions) DAD, etc. was an eye opener for me!

Youtube… ffs

For the longest time i’ve been using a Safari Extension called ClickToPlugin, which replaced Youtube’s video player with the native Safari video player. There were a couple of reason for this, the biggest of which was the horrendous amount of CPU that the YouTube HTML5 player uses. It also disabled autoplay, another scourge of the ad-supported web. Oh, and it never played ad.

The recent re-design broke all this, and it doesn’t look like it’ll be repaired. Time to find another solution… <sigh>

There are other Youtube focused extension out there for Safari, but none of them seem to exactly what i want. Firefox has a few plugins to allow downloading, or copying the video URL, which gives you a way to choose the player. There doesn’t, however, seem to be anything that does exactly what ClickToPlugin managed.

For a few weeks i’ve been using a Firefox plugin to copy the video URL, pasting that into Safari, and letting it play it with the native player. But it means opening Firefox, and switching between browsers, etc.

More recently i started playing with youtube-dl. If i’m going to copy and pasting URLs why not give them to a script, and have it spawn a Quicktime player? Well, the Quicktime player doesn’t have a command line… and who wants to wait until a video has downloaded before watching? It would be better to pipe the output of youtube-dl to a player… but that will have to be something other than Quicktime.

When in doubt try ffmpeg – the true swiss army knife of video! The ffmpeg distribution includes a tool ffplay, which can play video piped into stdin. Looks like we have everything needed:

$ youtube-dl -q -f best --buffer-size 16K https://www.youtube.com/watch?v=DLzxrzFCyOs -o - | ffplay -fast -fs -loglevel quiet -

Now all i need is a dumb bash script in my path, which takes a URL, and plugs it into that command:

#!/bin/bash
if [ $# -ne 1 ]; then
    echo Usage: yt url
    exit 1
fi

url=$1

youtube-dl -q -f best --buffer-size 16K $url -o - | \
 ffplay -fs -loglevel quiet -

Yes, the amount of time and effort involved in avoiding the unavoidable smartness of the smartest people in Silicon Valley…

Signal Desktop (Again)

It could be that i’m still confused about the release channel for the standalone Signal-Desktop client, but it doesn’t appear to be released as a bundle.

My previous build from the ‘electron’ branch stopped working, telling me i needed to update. However, the electron branch has gone… which is actually good news, as it means that the changes have merged to master.

Starting from scratch, but with all of the NodeJS NPM / Yarn junk still around, all it took was cloning from GitHub:

$ git clone –recursive https://github.com/WhisperSystems/Signal-Desktop.git

$ cd Signal-Desktop/
$ yarn pack-prod

Edit: the module dance can be avoided with the following:

$ node –version
v8.6.0
$ yarn –version
1.1.0
$ git clone –recursive https://github.com/WhisperSystems/Signal-Desktop.git

$ cd Signal-Desktop/
$ yarn install
$ yarn pack-prod

And then the usual dance to add modules until things started working. That part of the process defies description, and short of trying re-install all the npm / yarn stuff and starting from nothing, its unlikely that we’ll see the same things. In my case i had to ‘npm install’ the following:

  • electron-icon-maker
  • grunt

and ‘yarn add’:

  • grunt-sass

I’d have thought that there is a actually a module dependency list in the bundle, and a yarn / npm command to process it… no idea what that might be!

signal-desktop

It would be nice if there was an official build soon. Would like to stop having to do this dance – especially as the builds have been working perfectly for what i need for months now!

… No!

Last weekend / week some valiant efforts were made to import a Blogger export file into the latest WordPress release running on my laptop. Just getting that shitshow on the road was not a walk in the park. I’ll leave the next paragraph as a warning to future travellers on macOS:

WordPress will not connect to MySQL / MariaDB on localhost, update your config to use 127.0.0.1.

How PHP manages to be so consistently bad is a source of amazement.

My initial excitement about having found a plugin was tempered by the eventual realisation that the hooks are only there when saving a new entry. Presumably the code could be re-used in a pipeline that uses the Blogger REST API to pull entries, pushing them into WordPress, attaching images and rewriting img tags as it goes.

My hopes of getting this done without writing my own toolchain are slowly fading into the ether…

Just

You do it to yourself, you do
And that’s what really hurts
Is you do it to yourself, just you
You and no-one else
You do it to yourself
You do it to yourself

That’s the thing, if i was better at ignoring things…

Last weekend’s adventure was making a local backup of all my images in Flickr. This was really about getting to a point where it wouldn’t matter if that account got closed, or the new owners (Verizon?) decided to move everything, etc. It was a good start, but there is another (bigger) problem that is going to stop me from moving forward: the IWTFY blog.

There are hundred… i don’t know, maybe thousands of posts there that link directly to the images hosted on Flickr. Oops. Besides that being against the terms of service it rather neatly ties me to keeping the Flickr account for as long as there is an IWTFY blog… and that’s ongoing for a decade at this point!

Obviously a plan is needed!

  1. Export IWTFY from Blogger / Google – this seems straightforward.
  2. Setup a local instance of WordPress where i can run plugins.
  3. Write a plugin that pulls all of the images into the local WordPress instance
  4. Export IWTFY from the local instance
  5. Import into a hosted WordPress

This feels very much like a process that someone else must have already been through, and either made available / sells as a service… right? Proceeding onwards assuming that i’m not able to find that person / service, it looks like it isn’t difficult to use something like  the WP_Http API call to grab images from within a blog post. And somewhere in here there has to be a way loop through all the entries in blog. No idea what kind of WordPress *thing* i’m supposed to be coding, but yeah… Bildung!

Can’t actually believe i’m going to do this. Save me from myself? Please.

[Yes, the tags are something of a honeypot.]

Edit: someone already wrote the plugin i need! Import External Images – even if it doesn’t do exactly what is needed, its only a little PHP hacking… how bad could it be?!

Signal Desktop (Update)

My weekend adventures got me as far as a running Election branch. Something was obviously a little weird as it was opening with a javascript debugger taking up the right third of the app window. Not ideal, but easy to close.

And then i tried to link it to the account on my phone… no dice.

What i’d built was the development version, which defaults to connecting to the test / staging servers. For any of this experiment to be useful it needs to connect to the production servers.

These are all defined in the /config directory, but it wasn’t obvious to me how to change to another config. In the package.json there were references to ${env.SIGNAL_ENV} but exporting that to the shell with a value of ‘production’ didn’t help.

A little reading about my new friend ‘yarn’ showed that there was a ‘–production=true’ CLI option. Passing that along with ‘yarn install’ broke things.. but the command:

$ yarn start --production=true

Moves things along and makes it possible to link to a phone account. The process is actually pretty smooth – you just take a picture of a QRCode, give the client a name and you’re done.

What i’ve still not worked out is how to tell my other new friend ‘electron-packager’ to package the production version. Until that is worked out i’m stuck with the Muon version… which isn’t a bad place – it has been very stable thus far.

Edit: It seems ‘electron-packager’ was not my friend after all, ‘yarn’ however i’m starting to like! The ‘package.json’ contains a ‘scripts’ section defining a bunch of targets, one of which ‘pack-prod’ is ‘SIGNAL_ENV=production npm run dist’ which seems exactly what i need. And, indeed:

$ yarn pack-prod

spat out a ‘dist’ dir containing a working ‘production’ Signal.app.

The end.