Create nice utility apps Nim - as little code as possible

automation
data_science
javascript
nim
cross-platform
c
Tags: #<Tag:0x00007f5334f418a0> #<Tag:0x00007f5334f41760> #<Tag:0x00007f5334f41620> #<Tag:0x00007f5334f414e0> #<Tag:0x00007f5334f413a0> #<Tag:0x00007f5334f41260>

#1

Rapid utility development for data analysis

Nim? Another programming language in 2017? Is that really necessary? And Rapid Development? Who cares about that?

Nim is a compiled, garbage-collected systems programming language with a design that focuses on efficiency, expressiveness, and elegance (in the order of priority).

Ok, so it’s the perfect programming language? Who knows… but I have use cases for Nim.

##Summary:

Nim shares some characteristics with Python

If I need a quick GUI utility, I often use Python with PyQT. The development workflow is straight forward: I can use Qt Designer, a GUI builder.

This is one aspect of Rapid Application Development (RAD). This is fast, versatile, and straight forward - GUI driven development, to a degree. The combination of GUI builders and efficient programming languages, like Python, is popular.

Many people know Qt and Python. Personally I don’t like spending time on code for utilities and internal applications. I am not a professional software developer. That being said, please take this article with a grain of salt, as usual.

I am a security engineer. Most security libs and tools are written in Python. Like Scapy, sqlmap, sulley, Paimei… IDA Pro uses Python… it’s everywhere in security tools.

But Python... but cPython...

cPython has some disadvantages. When I refer to Python in the following, I mean cPython.

  • Python’s support for multi-threading and concurrency is limited due to the GIL. IronPython or Jython do not have this problem. But not all libraries are compatible to their runtimes.
  • Python 2 and Python 3 are 2 different beasts with different grammars. There are code transition utilities, but I found the library chaos annoying.
  • getting an executable (.exe, .bin) from Python projects is not a clean process. There are build utilities for this though, which may work. Or not.

I was searching for something else. No, not the “perfect programming language”. But something that has better performance, support for multi-core architectures, a nice development community and good language features.

Recently I discovered Nim, which has great potential. Nim is a programming language, which gets compiled to C, C++ or JavaScript.
This is a very unusual compilation model, and this makes a difference. Nim can automatically generate C code.

Nim is fast

If I need an analysis app, which takes 4-10 GB of local and or remote data in, and needs to create some output, Nim is fast. The performance is comparable to C or Rust.

  • Nim has issues, of course. The C code Nim generates is not easy to read. If there is a bug, we have to fix it the hard way. And C is the hard way.
  • Nim is a very young programming language. The development started in 2005.
  • If I am not mistaken Nim’s compiler options can influence the security of an application, because -d:release removes implicit bounds checks. That’s weird, because normally you’d expect that this strips debug symbols. If you lose bounds checking this way, you may be in trouble.

Secure coding in Nim

I am very interested in these aspects of Nim, because they show that secure software development can be supported by a compiler.

  • Nim is statically typed, and uses (lazy) type inference to resolve types. There are dynamic type checking features, such as runtime type information, which allows for dynamic dispatching of functions.
  • Nim automatically adds bounds-checks at compile- or run-time, to prevent buffer overflows.
  • It supports memory-safe code practices, but you can use pointers.
    • But pointer arithmetic is not possible for reference types since they are entirely managed by Nim’s garbage collector. This can prevent issues such as dangling pointers and other memory issues related to managing memory manually. In 2017 I have no plans to manage memory in code, if I can avoid it. Especially not for utility apps, or RAD.
    • But the garbage collector in Nim is optional! If you want to mess with it, you can.
  • variables are always initialized to their default values
  • Nim supports Generics, which is very useful for Parsers and OOP

That’s quite a feature list, isn’t it? That being said, I like Nim. It’s OpenSource, and it has got a friendly small development community.

Summary: Ni Na Nimda Nim

Nim is an innovative new programming language, also with security minded features. Since it can get compiled to C (and other languages), applications can be fast and efficient.

  • Programs can be multi-threaded and utilize multiple cores.
  • Apps can be shipped as an executable release.
    • No need for a runtime environment or an interpreter
  • It can use things like Qt, DataFrames, REST API calls - more on that later
  • It’s cross-platform, at least to a degree
  • Nim can be used to specify backends, which process a lot of data fast.
    • Real-time log enrichment? JSON parsing? Nim is your friend.
  • Nim supports OOP style coding, which I prefer. Nim supports common OOP features, including inheritance, polymorphism, and dynamic dispatching.
    • But in opposite to Java it doesn’t enforce it, which is good imho.

The reality is that often we don’t have enough time for software development. And for internal tools it’s even worse. I see Nim’s future at prototyping and RAD, GUI driven developent and in PoCs for new security tools.

Hands on: installation on Windows / Linux

I demo this on Windows, because Nim is supposed to be cross-platform. It works on Linux. Don’t you worry. You favorite package manager will set is all up for you. Windows is extra work.

Personally I am not into Linux-only software development. With a high-level programming / specification language like Nim it really shouldn’t matter.
I focus on Windows here, because, because tracking dependencies on Windows it not as easy as it could be. I have the feeling that I would have benefited from an article like this.

We need :

  • a recent Nim release in the PATH.
C:\nim\>nim -h
Nim Compiler Version 0.17.0 (2017-05-18) [Windows: amd64]
Copyright (c) 2006-2017 by Andreas Rumpf
  • a fitting MinGW (comes with the Qt installer) in the PATH. This is for 64 bit compilation.

  • preferably a Visual C++ compiler (Community Edition is enough) on Windows 8.1 or Windows 10.

C:\nim\nimdataframe>which gcc
/c/mingw/mingw64/bin/gcc
C:\nim\nimdataframe>gcc --version
gcc (x86_64-posix-seh-rev2, Built by MinGW-W64 project) 6.3.0
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  • Qt 5.8, including Qt Designer (free, OpenSource)

    • I use Qt for GUI development. Nim has Qt bindings. - More on that later, but for Qt you may want Visual C++ compilers. Unless your compilation skills are awesome. Qt does not release for MinGW 64.
  • Git, cmake and some dev stuff - you can see that I use some GNU tools on Windows . Get Cmder and Chocolatey. You know your dev tools better than I do.

Windows - just ported NimCx

I ported NimCx to use the nimdataframe code on Windows. In the diff you can see that it’s 4-5 changed lines of code, for now. This is work in progress, but I like to start like this.

Demo:

I think this is about Aliens?!

Cmder here can use colors in a terminal. I don’t see why Windows needs to be excluded. Sure. cmd.exe might not do this properly. But there are many virtual terminal implementations which suck on all kinds of operating systems.

Nim and JSON - examples

The Nim json module isn’t as easy as Python’s with dicts. I think for some demo code this is fine. Don’t take this as a reference code, but as an example to follow along and to get a feeling for Nim.

I am not a professional Nim coder, but this works for me. I am going to get better at this, and this will evolve.

proc get_url() : string =
    return "C:\\nim\\sample\\train.json"

Ok, the windows separator needs to be escaped, but that is all. This function / proc returns a string.

proc main() : void =     
    # get and parse file
    var file = get_url()
    var json_data = file.parseFile()

The main() returns nothing (void). You could do this in a one-liner, but this is just a call on parseFile(). No big deal here. Coding and feeling.

# retieve the keys and sort them
let key_list = get_json_keys(json_data).sorted(system.cmp[string])
  • So… as a JavaScript coder you probably guessed that var is a keyword to define a variable. let does the same, but the the variable cannot be reassigned. It’s like final in Java. A const would be static.
  • json_data here holds the parsed JSON data. Here we want the keys. The module does not have a nice get_keys() like Python’s dict. That is bad, but actually not a problem.
  • The .sorted() call uses a comparison method from the algorithm module.

Ok, so what is in this method?

proc get_json_keys(json_data: JsonNode) : seq[string] =
    #[
        Get a list of keys for the values in the JSON
    ]#
    var key_list: seq[string] = @[]
    
    for record in json_data:
        for key, value in record:
            if (not key_list.contains(key)):
                key_list.add(key)

    return key_list
  • JsonNode is a container of JArrays (like json_data here). JArray's hold JObjects (like record here), which have the JSON key / value pairs.
  • This method returns a sequence of strings.

So, is that straight forward code? Two loops, couple of assignments and some syntax, which doesn’t look too crazy. The only thing which is not straight forward is the hierarchy of the objects, which you need to understand in order to access the JSON data.

The next thing I want to do is to have a table (with columns). I want to feed the JSON data into a table. Excel like. The code is very similar. First, this gets invoked:

# for each key, create a sequence for values in a table
var col_table = initOrderedTable[ string, seq[string] ]()
col_table = get_json_table(json_data, key_list)
  • this uses OrderedTable from the tables module
  • it passes the JsonNode container and the string sequence (key_list) with the keys

The code is straight forward as well:

proc get_json_table(json_data: JsonNode, key_list: seq[string]): OrderedTable[string, seq[string]] =
    #[
        Provide a common data structure with methods for the JSON data
    ]#
    # for each key, create a sequence for values
    var col_table = initOrderedTable[ string, seq[string] ]()

    for key in key_list:
        echo "Working on column: ", key
        echo ""
        var tseq: seq[string] = @[]

        doAssert(json_data.kind == JArray)

        for node in json_data:  
            doAssert(node.kind == JObject) 
            let entry = node[key].getStr().strip()
            if entry != "":
                tseq.add(node[key].getStr())
            else:
                tseq.add("Nil")
        
        col_table[key] = tseq

    return col_table

The syntax highlighting is… bad here. But I hear that’s going to change.

  • this iterates over the key list, and uses each key to generate a column in an OrderedTable
  • tseq is a temporary string sequence which gets build up. Each key gets its column as a sequence of strings
    • if there is no real data the entry will be Nil. The strip() is from strutils
  • The doAssert()s are something you are going to see in Nim quite a lot. This

Again the only strange thing is that you need to iterate over the data once, to get the keys. And another time to build up the columns. There are probably ways to do this differently, but the point here mainly is: this is how Nim code looks. This is how control flow, loops and variables work. This is how you define methods and return values. How you open a file, how you parse some data… And here is how you do something with the data.
Remember that we invoked the get_json_table() and returned the result into col_table:

# search table by key and count the amount of elements
echo col_table["requester_username"].len()
  • We go the column of “requester_username” and count the amount of entries. We could exclude the Nils, but I am sure that I don’t need to demo this.

How to make an API request?

This is actually really easy:

let response = make_api_request().parseJson()
echo response.pretty

echo "The title is: ", response["title"].getStr()

And the method:

proc make_api_request() : string =
    var client = newHttpClient()
    var response = client.getContent("https://jsonplaceholder.typicode.com/posts/1")
    return response
  • the .parseJson() methods returns a JObject after parsing the results from make_api_request()
  • response["title"].kind would tell you that the title key resolves a JString

Results: Parsing JSON with Nim is … okay, but a get_keys() method would be useful. The examples in the documentation are schoolbook examples. That’s annoying. Many focus on building objects, and assume that you know what kind of data you have in the JSON file. Also the data structure has a complex hierarchy. That results into many iterations.

Making API requests is no problem. Just remember to invoke the compiler with SSL, like nim c -d:ssl json_p.nim.

But this is a real world example. Of course Nim is not perfect.

Qt - nimqml and DLL hell

Time for some GUIs, for pretty things. Things which aren’t complex. We need more GUI apps, which are useful. Everywhere.

In order to avoid all kinds of problems you need to be clear about what you are doing:

  • check your PATH with RapidEE - are there Qt DLLs in there? 010 Editor? Anaconda? Other tools?
  • check your Visual Studio installation - C / C++ compilers. What is the version? Does the compiler you want to use fit the Qt release on your system?
  • Check your Nim installation - does it fit? I use x86_64. You might use x86_32.

I assume you know Qt well.

Get your Qt environment for Nim QML

NimQML is released here. The installation is straight forward: with a nimble install nimqml you are done and setup. That’s what you have been looking for, isn’t it? :slight_smile:

  • Qt 5.8 isn’t released for Mingw 64. But it’s released for Visual Studio 2015.
    • For Visual Studio 2015 the PATH entry for Qt may resemble C:\Qt_dev\5.8\msvc2015_64\bin. You should select the fitting release in the Qt installer.
    • I use Qt 5.8 because the nimqml depends on DOtherSide
      • put the DLL from the ZIP release in the directory of the compiled executable, or in the PATH
      • it is possible to build DOtherSide with MinGW. I have added this as a reference, based on this guide for D.
      • copy the fitting Qt 5.8 DLLs in the directory of the compiled executable, or in the PATH (see below)

Compile Nim projects with Visual Studio

Here is how I initiate the Visual Studio compiler from the command line. The nim compiler utility actually may call the script automatically.

C:\nim\nimql\examples\helloworld>"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64\vcvars64.bat"
C:\nim\nimql\examples\helloworld>which cl
/c/Program Files (x86)/Microsoft Visual Studio 14.0/VC/BIN/amd64/cl

In order to compile code with the Visual Studio compiler instead of GCC, you need to check your Nim configuration. Mine resides in C:\nim\nim-0.17.0\config\nim.cfg. If you open the file in an editor you will see a statement like this:

# You may set environment variables with
# @putenv "key" "val"
# Environment variables can be accessed like so:
#  gcc.path %= "$CC_PATH"

cc = gcc

That means that the default C / C++ compiler is GCC, on Windows.

You can override this default, but you need to keep in mind that even though you loaded the Visual Studio compiler in the PATH, Nim will by default use the compiler which is specified in the configuration. That is the usual behavior for many build tools.

I prefer to use the command-line override:

C:\nim\nimql\examples\helloworld>nim --verbosity:2 --cc:vcc c main
C:\nim\nimql\examples\helloworld>file main.exe
main.exe: PE32+ executable (console) x86-64, for MS Windows

With that verbosity level you will see, that the compilation process reads the nim.cfg, and then issues the compilation command based on the settings.

Understand DLL dependencies of compiled projects

Here you can see that I copied the DLLs over. That may not be necessary. But I had problems with the Windows DLL hell. Again this is a real-word example. Not a schoolbook :slight_smile:

C:\nim\nimql\examples\helloworld>dumpbin.exe /imports DOtherSide.dll | grep dll
Dump of file DOtherSide.dll
    Qt5Widgets.dll
    Qt5Quick.dll
    Qt5Gui.dll
    Qt5Qml.dll
    Qt5Core.dll
    MSVCP140.dll
    VCRUNTIME140.dll
    api-ms-win-crt-runtime-l1-1-0.dll
                          3F _seh_filter_dll
    api-ms-win-crt-heap-l1-1-0.dll
    KERNEL32.dll

The Qt DLLs should reside in the same directory, to avoid issues and to be able to package the release.

C:\nim\nimql\examples\helloworld>ls -al *.dll
-rwxr-xr-x 1 marius 197121  767488 Jul 14 08:30 DOtherSide.dll*
-rwxr-xr-x 1 marius 197121 5663744 Jul 14 09:37 Qt5Core.dll*
-rwxr-xr-x 1 marius 197121 5981184 Jul 14 09:37 Qt5Gui.dll*
-rwxr-xr-x 1 marius 197121 3207680 Jul 14 09:37 Qt5Qml.dll*
-rwxr-xr-x 1 marius 197121 3349504 Jul 14 09:37 Qt5Quick.dll*
-rwxr-xr-x 1 marius 197121 5510656 Jul 14 09:37 Qt5Widgets.dll*

You might see popups which tell you that certain methods were not resolvable within Qt DLLs. Then you may need to remove these DLLs from the PATH.
Yes, this is DLL hell :wink: I solve it by having these kinds of DLLs next to the resulting executable, so that I can package the software into an installer. Zip, pack and ship.

The Qt Designer workflow with QML and Nim

In the same directory you can find a QML file, which can be edited with Qt Creator (Community Edition).

Results: This is a good start. You can design the GUIs, save the QML, and write Nim code around it.
Windows can be harsh with the PATH and the DLLs. Qt does not have a release for MinGW 64. If you need that, you need to build Qt. I chose to use the Visual Studio compiler instead. Nim works well with it.
This way very little code (Nim is condensed like a high scripting language) can result into a GUI app. Sure, this is just a small app. But it’s enough for a tutorial.

Build DOtherSide with MinGW and Cmake on Windows 10

For the QML bindings in Nim you may want the DOtherSide DLL, built for MinGW. That’s when your cmake installation comes in handy:

C:\nim\DOtherSide\build>cmake -G "MinGW Makefiles" ..
-- The C compiler identification is GNU 6.3.0
-- The CXX compiler identification is GNU 6.3.0

Next is make :

C:\nim\DOtherSide\build>mingw32-make.exe
Scanning dependencies of target DOtherSide_autogen
[  3%] Automatic MOC for target DOtherSide

The next thing would be to find or build a fitting Qt release.

My recommendation is to rely on Visual Studio releases. You might be lucky, and the releases fit. But luck sometimes runs out at certain moments :wink:

A Single Page App with Nim

So, let’s just create a nice Single Page (web) App (SPA). We could throw this into a Qt Web Frame. Single Page - Single Window - Single executable - simple app.

For Nim there is karax to generate SPAs. But at the moment nimx has more momentum, and therefore let’s use this.

The scrollapp scample can be compiled with nim js scrollapp.nim. This will generate JS code, which is not readable for humans, but which is usable from HTML. The HTML gets built with a Nim method. Karax starts with setRenderer, and adds the event handlers with the DOM.

In a browser the page looks like this:

Results: it’s possible to build SPAs with Nim and karax. I would not do this.

Web app development with Nim!?

There is Jester, which looks promising to me. For simple apps, forms and routes. Throwing some quick CSS around it, basic JS - sounds doable.

But I don’t develop Web apps with Nim for now. If I need a quick one-shot web app I can use:

  • Python Flask (which does so many things) with Bootstrap (via a Wireframe tool, like this (€))
  • just a Spring Micro-service (bare, REST) - this is a new paradigm and not the complex dependency-hell dev env style Java Enterprise development you might associate with it

JQuery, Ember.js or Angular.js are nice, but not Rapid. It’s a lot of work to integrate these heavy JS libs properly.

(Other) rapid development ways I used so far

As I said, I think Nim’s use cases for me are in RAD, prototyping, utility apps and maybe high-performance backends.

RAD Studio

  • For commercial development RAD Studio was pitched to me once.
    • You can have C++ projects, and go for cross-platform / cross-device UI development.
      • Qt Creator and Qt hold well for many complex software products
    • I have used RAD Studio with Boost, and Python. That of course limits the portability.

It doesn’t look like RAD Studio has the right momentum for Web App development. Just my 2 cents: I don’t want to develop Web Apps in C++. Who does that?!
Also I don’t see how I can park their proprietary fat server in a Docker container to host it cheaply. Granted, that may not be necessary. But it indicates that this is clumsy.

RAD Studio can be useful to work with clients, to derive specifications little by little. But that usually isn’t my line of work.

Summary: We need more Nim in our life

Because more Nim means better Nim! There isn’t too much Nim(rod) code out there, which can be used as an example.

  • For cross-platform command-line apps Nim works well.

    • Due to C the resulting executables can be fast. They might need DLLs or SOs to work, and there can be issues. Generally speaking: it’s worth it, if you want less Python and more performance. And these “issues” are not Nim specific.
  • For cross-platform Nim-based Desktop GUIs Qt is promising. The apps will have some native look and feel, and require little coding. Single-page web apps seem to be a thing I’d do in Nim with nimx. Or not at all :slight_smile:

  • For fast-forward Web App development non-Nim based frameworks are more versatile.

    • I’d like to see some more development there. Let’s see how much time I have. These days everything needs to be a web app. Desktop apps are so early 2000s…
  • For backend libraries, being able to wrap binding interfaces into Python is very nice. This adds a lot of flexibility, which is very important before you commit your time into something like Nim.

    • There might be issues, and quirks. These can eat a lot of time. And you may need a workaround.
    • I like using Nim as a backend, behind the scenes.
  • For Cloud-apps I’d like to see more of Nim. - If a small container with a compiled executable counts as Cloud-native development.

    • Modern Backend as a Service (BaaS) / Server-less architectures often specify REST endpoints, and wrap some libraries around the calls. Every capable language can interface with this.

Thing is, that you need to be careful, that your prototypes don’t get too good. Otherwise you won’t get the project to develop the “real app”.

At the moment Nim’s documentation has too many schoolbook examples and not enough real world solutions. The JSON module is not easy to use.

Things I wanna do - check this out bro

I WANNA ROCK! Apart from that:

Wrapping a Kivy GUI - let Nim do the heavy lifting. View - Backend

A promising Python project is Kivy, and they are working on a GUI builder. We can just create a heavy lifting module in Nim, wrap it to become a Python extension, and make use of Kivy for GUI apps.

References
Version history

22.07.2017 - initial release
23.07.2017 - added summary links, fixed some headings
04.08.2017 - added reference about debugging with Nim with GDB