Resuming uploads to Amazon S3

I am escaping continental winter at the moment which means my internet connection is not perfect. I was uploading a 1GB file to Amazon S3 via the and the upload failed at 93%. Just my luck!

I didn’t want this to happen again. So I tried using Cyberduck to upload and intentionally turned off WiFi after a few seconds to see if I can resume the upload. Nope. Does not work. OK, what about the official AWS command-line tool? Also nope. Oh, come on!

Finally, I found this StackOverflow entry. Apparently `s3cmd` does support resuming uploads. Let’s give it a try:

~$ s3cmd put --multipart-chunk-size-mb=5 IRL_talk_recording.mp4 s3://
upload: 'IRL_talk_recording.mp4' -> 's3://'  [part 1 of 211, 5MB] [1 of 1]
  5242880 of 5242880   100% in   45s   111.64 kB/s  done
upload: 'IRL_talk_recording.mp4' -> 's3://'  [part 2 of 211, 5MB] [1 of 1]
  5242880 of 5242880   100% in   37s   136.04 kB/s  done
upload: 'IRL_talk_recording.mp4' -> 's3://'  [part 3 of 211, 5MB] [1 of 1]
  5242880 of 5242880   100% in   25s   204.51 kB/s  done
upload: 'IRL_talk_recording.mp4' -> 's3://'  [part 4 of 211, 5MB] [1 of 1]
  524288 of 5242880    10% in    3s   155.49 kB/s^CERROR:
Upload of 'IRL_talk_recording.mp4' part 4 failed. Use
  bin/s3cmd abortmp s3:// zOidfC...5RHKek_DS
to abort the upload, or
  bin/s3cmd --upload-id zOidfC...5RHKek_DS put ...
to continue the upload.
See ya!

~$ s3cmd --upload-id zOidfC...5RHKek_DS  put --multipart-chunk-size-mb=5 IRL_talk_recording.mp4 s3://
WARNING: MultiPart: size and md5sum match for s3:// part 1, skipping.
WARNING: MultiPart: size and md5sum match for s3:// part 2, skipping.
WARNING: MultiPart: size and md5sum match for s3:// part 3, skipping.
upload: 'IRL_talk_recording.mp4' -> 's3://'  [part 4 of 211, 5MB] [1 of 1]
  5242880 of 5242880   100% in  230s    22.17 kB/s  done
upload: 'IRL_talk_recording.mp4' -> 's3://'  [part 5 of 211, 5MB] [1 of 1]
  2097152 of 5242880    40% in   60s    33.59 kB/s

Success! Glad I finally found this. Today is not the first time I got failed S3 uploads on flaky internet connections. Now I know how to resume them!

IRL#4: Istanbul

Last month we again held our biannual in-person gathering of all Niteans, the IRL. As is customary, we meet somewhere warm for the winter edition of the event. This year we chose Istanbul, Turkey. A stunning city with millennia of heritage.

We’ve gotten good at running IRLs so this one was by far the most productive. We’ve done several improvements to our policy and process (published in our Handbook), set good goals for the year and planned out the coming months. We had technical discussions, we had marketing brainstorms. We had coding sessions in the sun, we listened to lightning talks while sipping Mango Rum (thanks, Marbe!) and we took turns trying to beat Dejan at “foosball”. With a ton of fantastic food in between.


One of the outcomes I personally am most proud of is that we are charging ahead with our purpose, to help improve lives, using information and software as the main driving force. We are doing so by moving into a more mainstream market with our upcoming project so that we will be able to help “regular Johns and Janes” and not just hard-core internet marketers. There’s more: in the past, we have donated to great organizations such as LibreOffice, Wikipedia and Let’s Encrypt. Now, we’ve set it as a company goal for this year to make our impact on such causes an order of magnitude greater. In order to do that, we are 10X-ing our donations in 2018!

Since Niteo has grown a lot in the last year I finally got to meet some of the fellow Niteans in real life. It’s always nice to paint a face over that Slack profile. And go deep on some random topic with someone you only had text-based conversations before. If you work in a remote team, I heartily recommend getting everyone in the same location for a few days. It’s pure magic.

AWS EC2 gotchas

Lately, AWS has been giving us a lot of headaches on the Easy Blog Networks project. It turns out only because resources (we pay per how many things we can do per second) are fiercely measured on AWS, but the underlying problems are happening on all EBN providers. So when we go over operations per second limit which is 100, the EC2 hypervisor will limit entire server by lowering CPU resources. We see that as CPU steal. Now in order to figure out what exactly is using more than 100 operations per second, I installed netdata on a few servers. Which gave me the following picture:

In above picture we can see enormous spike in CPU usage by group of apache2 servers.

We had ~1.8k I/O operations(read,open,mmap syscalls), the limit is 100. And we still had a lot of credits to spare.

But AWS has this thing called burstable limit: you can have 100OP per second but you can also spike over that limit (3000 in our case). The burst is limited to 5-seconds moving average window if instance does not lower usage in that timespan, entire instance is punished by taking away CPU resources. One can disable this behavior on all x2 instances (Enable T2 unlimited), but this will incur additional costs, if you have constant usage above limits and not just spikes.

Now what that means is that some scripts are doing more things than anything else (backups don’t even come close to this). Tracking it down I came to something running mysql query:

The process was up for almost 5 minutes, where it can only be up for 90 seconds. To get things in order I limited MySQL runtime from 28k seconds which is the default, to 30 seconds.


This is after the lowered runtime limit was applied:

I’m still tracking down what exactly is causing extremely expensive queries because when the server gets spanked by AWS, every query is considered “costly” and then no monitoring helps. But if we catch it just before it happens, we have the culprit! 🙂

To be continued …

Recording Talks

Last year on IRL#2, our biannual in-person get-together, we decided to start recording talks, so that those that could not attend in person would be able to watch them at some later point in time. Turns out, there is another great use case for these recordings: onboarding. Whenever a new person joins our team, these talks help to get them up to speed. The recordings have been a huge success and we’ve started doing them regularly when we give a talk, at local meetups, at conferences without dedicated video teams, etc.

Our setup is opinionated based on what we need and what devices we already have. It tries to be as simple & portable as possible while keeping good audio quality:

  • An iPhone for video: video is not very important when recording talks and iPhone’s camera is good enough.
  • A fantastic external shotgun microphone for audio: we often have more than one speaker (at once or alternating), we have questions from the audience, we have remote attendees pitching in, etc. We wanted our setup to be invisible, without attaching microphones to speakers’ shirts, without passing microphones around for questions and without complicated channels mixing in post-production. It turns out, the Rode VideoMic Pro “shutgun” microphone does the job perfectly. We set it up 7 to 10 meters from the speaker, among the audience, pointed at the speaker. Since the microphone is directional, it picks up the speaker’s voice very well while also allowing the comments from the audience to get recorded with sufficient quality. Set and forget! Do note that you need the SC4 adapter to be able to use the VideoMic with an iPhone.
  • Use FiLMic Pro app on iPhone for best quality audio recording and audio monitoring via headphones.
  • A basic tripod with a custom smartphone mount so that we can mount both the iPhone and the mic on the same tripod, point both at the speaker and hit record.
  • Use Airdrop to send the video file to macOS and use iMovie for basic video editing. No cables involved there and iMovie’s intuitive UI makes the workflow as painless as possible.

Assuming you already have an iPhone and a tripod laying around, a rig like this will cost you around 200€, which is a bargain considering the quality and convenience you get:

  • 150€ Rode VideoMic Pro
  • 11€ Rode SC4 TRS-TRRS adapter
  • 13€ Rode SC6 Dual TRRS adapter
  • 17€ FiLMiC Pro app
  • 10€ cheap smartphone tripod mount
  • 10€ male and female 1/4 to 3/8 thread adapters

Custom smartphone mount

I was hunting around Amazon & eBay and could not find a similar thing that would ship fast enough before IRL#2, so I built one myself. I bought a “smartphone tripod mount” at a local gadget shop and glued in a female 3/8 to 1/4 tripod screw adapter so it has an additional tripod screw insert. Now I can screw the mount into a tripod, mount the iPhone and then screw in the VideoMic into the new insert I added to the top of the mount, using the male 3/8 to 1/4 tripod screw adapter. Works like a charm!

iMovie workflow

First, create images of all slides. If you have the presentation in Keynote, you can export as images. If you have it as PDF, you can use Automator to create images.

OK, fire up iMovie and let’s start:

  • import talk recording
  • start a new project
  • drag clip into the project area
  • click the “magic wand” icon to auto-tune audio and video settings
  • right click -> Show Clip Trimmer -> remove the noise in the beginning and end of the talk
  • insert slide image a few seconds after the slide is changed in the video, as “video overlay“
  • select all “video overlay” images, select “cutaway” and set opacity to ~80%
  • add first slide image to the beginning of the clip, add cross dissolve transition
  • add cross dissolve to the end of the clip
  • click File -> Share -> File to render the final video of the talk (Resolution: 720p, Quality: High, Compress: Better Quality)
  • use Handbrake to compress the video
    • Web optimized ticked
    • Format: MP4 File
    • Video Encoder: H.264
    • Framerate: 24
    • Peak Framerate (VFR) selected
    • Constant Quality selected and RF:20
    • Encoder Options: Preset: slow
    • Picture: Storage Size: 1280 x 720
    • Audio: Mixdown: Mono
    • Upload to (AWS S3 static website using


After recording 10+ talks we realized that things sometimes go wrong and it’s usually a human error: forgot to turn on the microphone, forgot to press record, etc. We now do the following to decrease the chance of not having a recording.

GoPro backup

Mount a GoPro to one of the tripod’s legs, using the standard GoPro pole mount. Stick a huge SD card in the camera, turn it on while setting things up and keep it running. If all else fails you at least get a backup recording.

Audio monitoring

With recording talks, it’s the audio quality that you should be most concerned with. Therefore it comes super handy to be able to monitor audio that is being recorded. This is how you do it:

  • Plug the Rode SC6 into your iPhone. Then plug the VideoMic + SC4 adapter to one of the mic inputs on the SC6 adapter. Finally, plug your headphones into the headphones output on SC6 adapter.
  • Use the FiLMic Pro app on iPhone to get audio passthrough, i.e. to hear in headphones what the microphone is recording.


Print out this recording checklist and run through it before every talk:

  • Enable Airplane Mode on iPhone.
  • Turn off all apps running on iPhone.
  • Plug the charger cable into iPhone.
  • Verify VideoMic is correctly plugged into iPhone.
  • Turn VideoMic on, move switch all the way to the right (“curved line” icon) to enable the high pass filter that reduces background noise.
  • Start FiLMic Pro app and verify its configuration (click the Overview icon):
    • 1280×720 @ Apple Standard
    • 24FPS / 24FPS
    • PCM @ 48.8 Kbps Mic: External Microphone
  • Lightly touch the surface of VideoMic. You should hear scratching in your headphones. If you don’t, then FiLMiC is using iPhone’s internal microphone and you should go a few steps back.
  • Ask someone to act as the speaker for a few seconds and try which volume setting on VideoMic works best for given conditions. The farther away the VideoMic is from the speaker, the more boost it needs (+20).
  • Start recording.
  • Stop recording.
  • Turn off the VideoMic.


Here’s a few seconds of a recent recording from IRL#4 in Istanbul.

Sample for Recording Talks blog post from Nejc Zupan on Vimeo.

Nitean Recognition

“Bang a gong, get it on.”

While the lyric is pulled from a modestly talented rocker in the 70s, it does hit a key entrepreneurial principle. Some companies and businesses ring a bell when something good happens or an achievement reached.

At Niteo, we give kudos and +1s.

The wall of happy

Laszlo Bock, head of Google’s People Operations, has a Wall of Happy outside his office where kudos are printed and put on a physical wall.

We took the idea and the bell tolling, and spin them into something that would work for our case.

Instead of a bell, we have a Wall of Happy where celebrations and peer recognition are documented. And since we’re a remote-first team, the wall-of-happy is a Slack channel.

The modality used to implement the Nitean recognition was based on the fact that Slack is our main and effective communication tool.

See our Handbook policy on this.

For what the bell tolls

The Nitean recognition aspires to keep the team motivated by creating a work culture where the simplest achievement is celebrated. Moreover, it also promotes higher peer engagement and better work communication. Both of which are key activities that underpin all aspects of business and life in one-way or another.

Whether it’s closing a sale, removing a blocker or helping out a fellow Nitean, the bell tolls.

There is no discrimination between achievements. From a modest to a momentous victory, the team recognizes and heralds it.

Creating a positive workplace culture

Having a multicultural team has a lot of pros to it. On the flip side, it could entail a lot more from everyone. Besides the work tasks and responsibilities, some adjustments are necessary for a conducive and convivial workplace. The Nitean recognition lets us celebrate the little wins as validations of the team’s progress and success. Because sometimes it’s the little things, the nuances that often get dismissed, that have the most impact.

There is so much more to running and being a part of a company than pure economics or technology decisions. A team with a good and fostering relations with one another is aces in most, if not all, company goals.