Rapid utility development for data analysis
Nim? Another programming language in 2017? Is that really necessary? And Rapid Development? Who cares about that?
Nim is a compiled, garbage-collected systems programming language with a design that focuses on efficiency, expressiveness, and elegance (in the order of priority).
Ok, so it’s the perfect programming language? Who knows… but I have use cases for Nim.
Nim shares some characteristics with Python
If I need a quick GUI utility, I often use Python with PyQT. The development workflow is straight forward: I can use Qt Designer, a GUI builder.
This is one aspect of Rapid Application Development (RAD). This is fast, versatile, and straight forward - GUI driven development, to a degree. The combination of GUI builders and efficient programming languages, like Python, is popular.
Many people know Qt and Python. Personally I don’t like spending time on code for utilities and internal applications. I am not a professional software developer. That being said, please take this article with a grain of salt, as usual.
I am a security engineer. Most security libs and tools are written in Python. Like Scapy, sqlmap, sulley, Paimei… IDA Pro uses Python… it’s everywhere in security tools.
But Python... but cPython...
cPython has some disadvantages. When I refer to Python in the following, I mean cPython.
- Python’s support for multi-threading and concurrency is limited due to the GIL. IronPython or Jython do not have this problem. But not all libraries are compatible to their runtimes.
- Python 2 and Python 3 are 2 different beasts with different grammars. There are code transition utilities, but I found the library chaos annoying.
- getting an executable (.exe, .bin) from Python projects is not a clean process. There are build utilities for this though, which may work. Or not.
I was searching for something else. No, not the “perfect programming language”. But something that has better performance, support for multi-core architectures, a nice development community and good language features.
This is a very unusual compilation model, and this makes a difference. Nim can automatically generate C code.
Nim is fast
If I need an analysis app, which takes 4-10 GB of local and or remote data in, and needs to create some output, Nim is fast. The performance is comparable to C or Rust.
- Nim has issues, of course. The C code Nim generates is not easy to read. If there is a bug, we have to fix it the hard way. And C is the hard way.
- Nim is a very young programming language. The development started in 2005.
- If I am not mistaken Nim’s compiler options can influence the security of an application, because
-d:release removes implicit bounds checks. That’s weird, because normally you’d expect that this strips debug symbols. If you lose bounds checking this way, you may be in trouble.
Secure coding in Nim
I am very interested in these aspects of Nim, because they show that secure software development can be supported by a compiler.
- Nim is statically typed, and uses (lazy) type inference to resolve types. There are dynamic type checking features, such as runtime type information, which allows for dynamic dispatching of functions.
- Nim automatically adds bounds-checks at compile- or run-time, to prevent buffer overflows.
- It supports memory-safe code practices, but you can use pointers.
- But pointer arithmetic is not possible for reference types since they are entirely managed by Nim’s garbage collector. This can prevent issues such as dangling pointers and other memory issues related to managing memory manually. In 2017 I have no plans to manage memory in code, if I can avoid it. Especially not for utility apps, or RAD.
- But the garbage collector in Nim is optional! If you want to mess with it, you can.
- variables are always initialized to their default values
- Nim supports Generics, which is very useful for Parsers and OOP
That’s quite a feature list, isn’t it? That being said, I like Nim. It’s OpenSource, and it has got a friendly small development community.
Summary: Ni Na Nimda Nim
Nim is an innovative new programming language, also with security minded features. Since it can get compiled to C (and other languages), applications can be fast and efficient.
- Programs can be multi-threaded and utilize multiple cores.
- Apps can be shipped as an executable release.
- No need for a runtime environment or an interpreter
- It can use things like Qt, DataFrames, REST API calls - more on that later
- It’s cross-platform, at least to a degree
- Nim can be used to specify backends, which process a lot of data fast.
- Real-time log enrichment? JSON parsing? Nim is your friend.
- Nim supports OOP style coding, which I prefer. Nim supports common OOP features, including inheritance, polymorphism, and dynamic dispatching.
- But in opposite to Java it doesn’t enforce it, which is good imho.
The reality is that often we don’t have enough time for software development. And for internal tools it’s even worse. I see Nim’s future at prototyping and RAD, GUI driven developent and in PoCs for new security tools.
Hands on: installation on Windows / Linux
I demo this on Windows, because Nim is supposed to be cross-platform. It works on Linux. Don’t you worry. You favorite package manager will set is all up for you. Windows is extra work.
Personally I am not into Linux-only software development. With a high-level programming / specification language like Nim it really shouldn’t matter.
I focus on Windows here, because, because tracking dependencies on Windows it not as easy as it could be. I have the feeling that I would have benefited from an article like this.
We need :
- a recent Nim release in the
Nim Compiler Version 0.17.0 (2017-05-18) [Windows: amd64]
Copyright (c) 2006-2017 by Andreas Rumpf
gcc (x86_64-posix-seh-rev2, Built by MinGW-W64 project) 6.3.0
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Qt 5.8, including Qt Designer (free, OpenSource)
- I use Qt for GUI development. Nim has Qt bindings. - More on that later, but for Qt you may want Visual C++ compilers. Unless your compilation skills are awesome. Qt does not release for MinGW 64.
Git, cmake and some dev stuff - you can see that I use some GNU tools on Windows . Get Cmder and Chocolatey. You know your dev tools better than I do.
Windows - just ported NimCx
I ported NimCx to use the nimdataframe code on Windows. In the diff you can see that it’s 4-5 changed lines of code, for now. This is work in progress, but I like to start like this.
I think this is about Aliens?!
Cmder here can use colors in a terminal. I don’t see why Windows needs to be excluded. Sure.
cmd.exe might not do this properly. But there are many virtual terminal implementations which suck on all kinds of operating systems.
Nim and JSON - examples
The Nim json module isn’t as easy as Python’s with
dicts. I think for some demo code this is fine. Don’t take this as a reference code, but as an example to follow along and to get a feeling for Nim.
I am not a professional Nim coder, but this works for me. I am going to get better at this, and this will evolve.
proc get_url() : string =
Ok, the windows separator needs to be escaped, but that is all. This function /
proc returns a
proc main() : void =
# get and parse file
var file = get_url()
var json_data = file.parseFile()
main() returns nothing (
void). You could do this in a one-liner, but this is just a call on parseFile(). No big deal here. Coding and feeling.
# retieve the keys and sort them
let key_list = get_json_keys(json_data).sorted(system.cmp[string])
var is a keyword to define a variable.
let does the same, but the the variable cannot be reassigned. It’s like
final in Java. A
const would be static.
json_data here holds the parsed JSON data. Here we want the keys. The module does not have a nice
get_keys() like Python’s
dict. That is bad, but actually not a problem.
.sorted() call uses a comparison method from the algorithm module.
Ok, so what is in this method?
proc get_json_keys(json_data: JsonNode) : seq[string] =
Get a list of keys for the values in the JSON
var key_list: seq[string] = @
for record in json_data:
for key, value in record:
if (not key_list.contains(key)):
JsonNode is a container of
record here), which have the JSON key / value pairs.
- This method returns a sequence of
So, is that straight forward code? Two loops, couple of assignments and some syntax, which doesn’t look too crazy. The only thing which is not straight forward is the hierarchy of the objects, which you need to understand in order to access the JSON data.
The next thing I want to do is to have a table (with columns). I want to feed the JSON data into a table. Excel like. The code is very similar. First, this gets invoked:
# for each key, create a sequence for values in a table
var col_table = initOrderedTable[ string, seq[string] ]()
col_table = get_json_table(json_data, key_list)
- this uses OrderedTable from the
- it passes the
JsonNode container and the string
key_list) with the keys
The code is straight forward as well:
proc get_json_table(json_data: JsonNode, key_list: seq[string]): OrderedTable[string, seq[string]] =
Provide a common data structure with methods for the JSON data
# for each key, create a sequence for values
var col_table = initOrderedTable[ string, seq[string] ]()
for key in key_list:
echo "Working on column: ", key
var tseq: seq[string] = @
doAssert(json_data.kind == JArray)
for node in json_data:
doAssert(node.kind == JObject)
let entry = node[key].getStr().strip()
if entry != "":
col_table[key] = tseq
The syntax highlighting is… bad here. But I hear that’s going to change.
- this iterates over the key list, and uses each key to generate a column in an
tseq is a temporary string sequence which gets build up. Each key gets its column as a sequence of strings
- if there is no real data the entry will be
strip() is from
doAssert()s are something you are going to see in Nim quite a lot. This
Again the only strange thing is that you need to iterate over the data once, to get the keys. And another time to build up the columns. There are probably ways to do this differently, but the point here mainly is: this is how Nim code looks. This is how control flow, loops and variables work. This is how you define methods and return values. How you open a file, how you parse some data… And here is how you do something with the data.
Remember that we invoked the
get_json_table() and returned the result into
# search table by key and count the amount of elements
- We go the column of “requester_username” and count the amount of entries. We could exclude the
Nils, but I am sure that I don’t need to demo this.
How to make an API request?
This is actually really easy:
let response = make_api_request().parseJson()
echo "The title is: ", response["title"].getStr()
And the method:
proc make_api_request() : string =
var client = newHttpClient()
var response = client.getContent("https://jsonplaceholder.typicode.com/posts/1")
.parseJson() methods returns a
JObject after parsing the results from
response["title"].kind would tell you that the
title key resolves a
Results: Parsing JSON with Nim is … okay, but a get_keys() method would be useful. The examples in the documentation are schoolbook examples. That’s annoying. Many focus on building objects, and assume that you know what kind of data you have in the JSON file. Also the data structure has a complex hierarchy. That results into many iterations.
Making API requests is no problem. Just remember to invoke the compiler with SSL, like
nim c -d:ssl json_p.nim.
But this is a real world example. Of course Nim is not perfect.
Qt - nimqml and DLL hell
Time for some GUIs, for pretty things. Things which aren’t complex. We need more GUI apps, which are useful. Everywhere.
In order to avoid all kinds of problems you need to be clear about what you are doing:
- check your
PATH with RapidEE - are there Qt DLLs in there? 010 Editor? Anaconda? Other tools?
- check your Visual Studio installation - C / C++ compilers. What is the version? Does the compiler you want to use fit the Qt release on your system?
- Check your Nim installation - does it fit? I use x86_64. You might use x86_32.
I assume you know Qt well.
Get your Qt environment for Nim QML
NimQML is released here. The installation is straight forward: with a
nimble install nimqml you are done and setup. That’s what you have been looking for, isn’t it?
- Qt 5.8 isn’t released for Mingw 64. But it’s released for Visual Studio 2015.
- For Visual Studio 2015 the
PATH entry for Qt may resemble
C:\Qt_dev\5.8\msvc2015_64\bin. You should select the fitting release in the Qt installer.
- I use Qt 5.8 because the
nimqml depends on DOtherSide
- put the DLL from the ZIP release in the directory of the compiled executable, or in the
- it is possible to build DOtherSide with MinGW. I have added this as a reference, based on this guide for D.
- copy the fitting Qt 5.8 DLLs in the directory of the compiled executable, or in the
PATH (see below)
Compile Nim projects with Visual Studio
Here is how I initiate the Visual Studio compiler from the command line. The
nim compiler utility actually may call the script automatically.
C:\nim\nimql\examples\helloworld>"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64\vcvars64.bat"
/c/Program Files (x86)/Microsoft Visual Studio 14.0/VC/BIN/amd64/cl
In order to compile code with the Visual Studio compiler instead of GCC, you need to check your Nim configuration. Mine resides in
C:\nim\nim-0.17.0\config\nim.cfg. If you open the file in an editor you will see a statement like this:
# You may set environment variables with
# @putenv "key" "val"
# Environment variables can be accessed like so:
# gcc.path %= "$CC_PATH"
cc = gcc
That means that the default C / C++ compiler is GCC, on Windows.
You can override this default, but you need to keep in mind that even though you loaded the Visual Studio compiler in the
PATH, Nim will by default use the compiler which is specified in the configuration. That is the usual behavior for many build tools.
I prefer to use the command-line override:
C:\nim\nimql\examples\helloworld>nim --verbosity:2 --cc:vcc c main
main.exe: PE32+ executable (console) x86-64, for MS Windows
With that verbosity level you will see, that the compilation process reads the
nim.cfg, and then issues the compilation command based on the settings.
Understand DLL dependencies of compiled projects
Here you can see that I copied the DLLs over. That may not be necessary. But I had problems with the Windows DLL hell. Again this is a real-word example. Not a schoolbook
C:\nim\nimql\examples\helloworld>dumpbin.exe /imports DOtherSide.dll | grep dll
Dump of file DOtherSide.dll
The Qt DLLs should reside in the same directory, to avoid issues and to be able to package the release.
C:\nim\nimql\examples\helloworld>ls -al *.dll
-rwxr-xr-x 1 marius 197121 767488 Jul 14 08:30 DOtherSide.dll*
-rwxr-xr-x 1 marius 197121 5663744 Jul 14 09:37 Qt5Core.dll*
-rwxr-xr-x 1 marius 197121 5981184 Jul 14 09:37 Qt5Gui.dll*
-rwxr-xr-x 1 marius 197121 3207680 Jul 14 09:37 Qt5Qml.dll*
-rwxr-xr-x 1 marius 197121 3349504 Jul 14 09:37 Qt5Quick.dll*
-rwxr-xr-x 1 marius 197121 5510656 Jul 14 09:37 Qt5Widgets.dll*
You might see popups which tell you that certain methods were not resolvable within Qt DLLs. Then you may need to remove these DLLs from the
Yes, this is DLL hell I solve it by having these kinds of DLLs next to the resulting executable, so that I can package the software into an installer. Zip, pack and ship.
The Qt Designer workflow with QML and Nim
In the same directory you can find a QML file, which can be edited with Qt Creator (Community Edition).
Results: This is a good start. You can design the GUIs, save the QML, and write Nim code around it.
Windows can be harsh with the
PATH and the DLLs. Qt does not have a release for MinGW 64. If you need that, you need to build Qt. I chose to use the Visual Studio compiler instead. Nim works well with it.
This way very little code (Nim is condensed like a high scripting language) can result into a GUI app. Sure, this is just a small app. But it’s enough for a tutorial.
Build DOtherSide with MinGW and Cmake on Windows 10
For the QML bindings in Nim you may want the
DOtherSide DLL, built for MinGW. That’s when your
cmake installation comes in handy:
C:\nim\DOtherSide\build>cmake -G "MinGW Makefiles" ..
-- The C compiler identification is GNU 6.3.0
-- The CXX compiler identification is GNU 6.3.0
Scanning dependencies of target DOtherSide_autogen
[ 3%] Automatic MOC for target DOtherSide
The next thing would be to find or build a fitting Qt release.
My recommendation is to rely on Visual Studio releases. You might be lucky, and the releases fit. But luck sometimes runs out at certain moments
A Single Page App with Nim
So, let’s just create a nice Single Page (web) App (SPA). We could throw this into a Qt Web Frame. Single Page - Single Window - Single executable - simple app.
For Nim there is karax to generate SPAs. But at the moment nimx has more momentum, and therefore let’s use this.
The scrollapp scample can be compiled with
nim js scrollapp.nim. This will generate JS code, which is not readable for humans, but which is usable from HTML. The HTML gets built with a Nim method. Karax starts with setRenderer, and adds the event handlers with the DOM.
In a browser the page looks like this:
Results: it’s possible to build SPAs with Nim and karax. I would not do this.
Web app development with Nim!?
There is Jester, which looks promising to me. For simple apps, forms and routes. Throwing some quick CSS around it, basic JS - sounds doable.
But I don’t develop Web apps with Nim for now. If I need a quick one-shot web app I can use:
- Python Flask (which does so many things) with Bootstrap (via a Wireframe tool, like this (€))
- just a Spring Micro-service (bare, REST) - this is a new paradigm and not the complex dependency-hell dev env style Java Enterprise development you might associate with it
JQuery, Ember.js or Angular.js are nice, but not Rapid. It’s a lot of work to integrate these heavy JS libs properly.
(Other) rapid development ways I used so far
As I said, I think Nim’s use cases for me are in RAD, prototyping, utility apps and maybe high-performance backends.
- For commercial development RAD Studio was pitched to me once.
- You can have C++ projects, and go for cross-platform / cross-device UI development.
- Qt Creator and Qt hold well for many complex software products
- I have used RAD Studio with Boost, and Python. That of course limits the portability.
It doesn’t look like RAD Studio has the right momentum for Web App development. Just my 2 cents: I don’t want to develop Web Apps in C++. Who does that?!
Also I don’t see how I can park their proprietary fat server in a Docker container to host it cheaply. Granted, that may not be necessary. But it indicates that this is clumsy.
RAD Studio can be useful to work with clients, to derive specifications little by little. But that usually isn’t my line of work.
Summary: We need more Nim in our life
Because more Nim means better Nim! There isn’t too much Nim(rod) code out there, which can be used as an example.
For cross-platform command-line apps Nim works well.
- Due to C the resulting executables can be fast. They might need DLLs or SOs to work, and there can be issues. Generally speaking: it’s worth it, if you want less Python and more performance. And these “issues” are not Nim specific.
For cross-platform Nim-based Desktop GUIs Qt is promising. The apps will have some native look and feel, and require little coding. Single-page web apps seem to be a thing I’d do in Nim with
nimx. Or not at all
For fast-forward Web App development non-Nim based frameworks are more versatile.
- I’d like to see some more development there. Let’s see how much time I have. These days everything needs to be a web app. Desktop apps are so early 2000s…
For backend libraries, being able to wrap binding interfaces into Python is very nice. This adds a lot of flexibility, which is very important before you commit your time into something like Nim.
- There might be issues, and quirks. These can eat a lot of time. And you may need a workaround.
- I like using Nim as a backend, behind the scenes.
For Cloud-apps I’d like to see more of Nim. - If a small container with a compiled executable counts as Cloud-native development.
- Modern Backend as a Service (BaaS) / Server-less architectures often specify REST endpoints, and wrap some libraries around the calls. Every capable language can interface with this.
Thing is, that you need to be careful, that your prototypes don’t get too good. Otherwise you won’t get the project to develop the “real app”.
At the moment Nim’s documentation has too many schoolbook examples and not enough real world solutions. The JSON module is not easy to use.
Things I wanna do - check this out bro
I WANNA ROCK! Apart from that:
Wrapping a Kivy GUI - let Nim do the heavy lifting. View - Backend
A promising Python project is Kivy, and they are working on a GUI builder. We can just create a heavy lifting module in Nim, wrap it to become a Python extension, and make use of Kivy for GUI apps.
22.07.2017 - initial release
23.07.2017 - added summary links, fixed some headings
04.08.2017 - added reference about debugging with Nim with GDB