Phil Dreizen

Announcing Meme It So! It's a Star Trek The Next Generation Meme and GIF Generator, inspired by Frinkiac.

It has every line of dialogue spoken on STTNG tied to screencaptures. Which makes it good for looking up lines as well as making memes.

This is the big project I've been working on for the past month! I think it's finally ready for general use.

memeit.so

As I fall further and further behind on writing these TIL posts, it is becoming very clear that there is a lot of friction around a blog post. I think there's a feeling that a post needs to be "complete" in some way, a publishable thing. And it's getting in the way. I'm looking for alternatives.

The past few years has seen the growth of so called Digital Gardens. They're essentially public personal wikis. I'm not yet sure how I'm going to integrate them here, but I can tell right away that using TiddlyWiki the last 2 days, it's much easier to write into it. So, I've added a digital garden link to the top here. It's implemented in TiddlyWiki.

Professionally, I work as backend developer on a project that uses React in the frontend. I do have experience working with our React code of course, but it's far from the main thing I do.

Since I'll be using React in a bigger personal project I'm working on I wanted to learn the fundamentals in a formal way. I took an intro React course on egghead.io. Though the course was fine, I have to say, I just think using a book or written tutorial just works better for me. The Road to React is one of a few well regarded books on React, and so I worked through all the chapters. The advice at the end of the book is good: just go write a react project. Don't do any more tutorials.

So here is my first react "projectlet" - an implementation Tic Tac Toe. Nothing fancy here, just a working version of tic-tac-toe. It might be fun to add in an AI opponent, but that really has little to do with React. And of course, there's plenty of room to improve the look.

FYI: I'm really falling behind on my TIL posts! Today I'm talking about something a few weeks old.

I mentioned looking in Flask-RESTful previously. It seems to be an abandoned project. A fork, Flask-RESTful is also abadoned. But a recent fork, Flask-RESTx does appear active.

A former developer of Flask-RESTful suggested some recommended alternatives in a github comment. Notably, MethodView as an alternative to Resource addresses one of the concerns I had regarding Flask.

In conclusion, I'll be sticking with Flask + libs instead of using Flask-RESTful.

I have several hundred gigabytes worth of very small files I'm transferring from my laptop to my NAS server via an NFS mount over wifi. You'd think copying some (okay, a lot) of files around would be trivial, but it turned out not to be so easy.

My first attempt was perhaps naive. I simply used mv:

> mv $src_dir $dir_on_nas

But it was taking forever. Many hours later I gave up and decided to try again with a tar pipe. I tried something like:

> cd $src_dirtar cf - . | tar xf - -C $dir_on_nas

I ran it overnight and it did not get very far. In hindsight, I'm not sure why I thought this would be much different then using mv.

In any case, my next step was to take a look at what my transfer speed was. TIL about the pv which can measure the progress of data through a unix pipe. If you provide it the size of the data being transferred via the -s argument, it can also provide an ETA. And so the above tar pipe was modified to:

tar cf - . | pv -s $(du -sb . | awk '{print $1}') | tar xf - -C $dir_on_nas

pv displayed a paltry transfer rate between 1-2 MiB/s. This is just frustratingly slow, and explained why everything was taking so long.

My next step was to measure the network speed between my laptop and the NAS. TIL about iperf3, which sets up a socket that listens on one machine; other machines connect to the socket via iperf3 to measure the connection speed. (I should note, my NAS is a Synology, so installing iperf3 required jumping through a few hoops itself.).

#on the NAS
> iperf3 -s

#on my laptop...xxx is a stand in for the NAS ip
> iper3 -c 192.168.x.xxx

My transfer speed is not good: ~25MiB/s, but a lot better than ~1. Clearly, I need to troubleshoot the LAN itself, but for now, I'm going to leave it alone. At this point, I'd be happy with that kind of speed.

From the beginning I did suspect that the problem could be that I'm transferring a lot of small files. But I didn't think it would have this much of an impact. Cursory searches confirm that this is a pretty big problem, and I've found plenty of advice about various configuration changes I could make to the mount command to make the transfer go faster. But, if the issue really is just the number of files, I'm satisfied to transfer them over as a tarball and extract it later.

Before trying the single file tar, I decided to use dd to measure write speed to the NAS. Something like:

dd if=/dev/zero of=$dir_on_nas/test.dat count=100000

This showed a transfer speed of about 23MB/s, which is much more in line with iperf3. That really does suggest that I should be able to achieve that kind of speed if I packed up my smaller files into just 1 file.

So, this time, I tried transferring the files without extracting them (and notice I'm compressing here as well):

tar -zcf - . | pv -s $(du -sb $s | awk '{print $1}') > "$dir_on_nas/file.tar.gz" 

And yes pv is showing speeds of 22-23 MiB/s! Yay!

Except I should mention: I also tried this w/o compressing the data (leaving off the -z flag). Without compressing, pv reports the same transfer speed, but seemingly the transfer would "freeze" every few seconds. I still can't quite explain it. It has the "feel" of the pipe getting filled and the tar process getting blocked until the pipe empties, but wouldn't pv report a lower transfer rate as as result? Maybe not? I'm not sure.

In any case, now, with a single compressed file, the transfer speed is much faster now which is great.