Adventures in Python-land

Background

A friend of mine approached me with the task of writing a program. Sometimes he gets a call and has to check if something's going on at another production site. He wanted a tool to download still images from a security camera, which should be converted to an animated GIF or video.

So far, so good. A simple task one might think. Now let's talk about his interesting requirements. The program should be standalone so he can distribute it via software management to another person's computer. He wants to enable those other person to run the program. The program itself should be standalone without external dependencies like FFMpeg (which was my first thought when I heard image creation). Hmmm...

I wanted to use Python for this as it allows me to quickly create a prototype and there are tons of available libraries to convert/create images and videos.

Creating a GIF

First draft

I started with two suggestions from StackOverflow: Pillow and imageio. Both are libraries to create videos from images. You only need about 10 lines of code to get results. The problem with this approach was the resulting program was HUGE. To create a standalone executable from Python sources I used PyInstaller. The cause for the file size is image libraries need about 60 to 80 MB. So this was no solution I could use.

Using other programming languages

I knew you could call other programming languages through Python's FFI. I don't like to write C by myself so I searched for a ready-to-use solution in Rust but had no luck. However, there's a solution in Go and I found an article explaining how to use Go to build a module importable by Python. The resulting code only took about 3 MB space, so it was a perfect substitution for the fat Python. Calling Go methods as described by Filippo didn't work out for me and so I asked a friend of mine, who is a Go guru. He had a look at Filippo's blog post, but couldn't figure it out either. He mailed me back the refactored Go code and how he called it by using Python's ctypes - thanks again, Basti!

Ok, we've got a solution to create a GIF from given images that has a small footprint.

To compile your Go code into a shared object which is usable by ctypes use:

go build -buildmode=c-shared -o libcreategif.so createGif.go

GUI

As if getting a working solution for image conversion wasn't enough I thought it would be best to create a GUI. I wanted to inform the user of the image downloading process, present a choice from which site to choose from and when the video was created a file dialog should pop up asking the user where to save the file.

wxPython

I wanted to have native controls so I chose wxPython as my first approach. I've already used Kivy once and got disappointed (turns out I'll never use wxPython again either). The documentation for wxPython is OK-ish and I came up with a GUI faster than I originally thought. I used a dropdown menu to choose which source to take and by using pubsub I was able to tell the main Python thread what the status of the downloading worker process was. My program was working as expected. At least until I tried to create a standalone with PyInstaller. Apparently PyInstaller doesn't like wxPython's pubsub mechanism. I trashed this pubsub solution and tried PyPubSub, but again: no chance of getting pubsub to work with PyInstaller. That day I talked to my Go friend and he told me he had similar experiences using pubsub for his master thesis and he wrote his own solution back then.

PyQt

I complained about my situation at my favourite social network and someone answered he hasn't got a good impression of wxPython either and instead I should use PyQt or toga. I chose PyQt as I've worked with Qt already. Ok, let's ditch wxPython and write the GUI in PyQt5 then. Another advantage of ditching wxPython was I could use Python 3 now, because wxPython does work with Python 2 only. The great tutorials enable you to come up with a small GUI easily, there's even an example of how to implement a QProgressBar to how to update its value. I changed my image downloader from a native Python thread to QThread, because it allows you to use Qt's signals and slots mechanism to communicate over threads.

Executable for Windows

Another problem solved. The code's working, now let's build the Windows executable. Move ahead, we got work to do!

Go needs gcc as compiler. The thing is I tried Cygwin, but Go wants to have MinGW instead. The parameters to build a DLL are a little different to what you would use on Linux:

go build -buildmode=c-archive createGif.go
gcc -shared -pthread -o libcreategif.dll createGif.a -lWinMM -lntdll -lWS2_32

If you run

pyinstaller --windowed --onefile /path/to/python_programm.py

there will be errors nonetheless. Apparently there's something wrong when using PyQt 5.7 and PyInstaller - the Qt DLLs won't be copied, which you can solve as described here. To tell PyInstaller to copy our Go DLL into the executable and to use Qt DLLs tweak your .spec-File to look like this:

# -*- mode: python -*-

block_cipher = None


a = Analysis(['src\\main.py'],
             pathex=['Python\\Python35\\Lib\\site-packages\\PyQt5\\Qt\\bin', 'D:\\holz'],
             binaries=[ ('src/HolzLib/libcreategif.dll', '') ],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          name='Picidae',
          debug=False,
          strip=False,
          upx=True,
          console=False , icon='camera.ico')

Task accomplished!

Summary

While it was a cool project for several nights I would do things differently now:

• Ask the PO to give me more information about the problem he wants me solved
• Ask for a VM or another machine where he wants to run the program
• CI/CD on that machine