Verify a method was called with certain argument using Mockito

- Last update:

Unit test are important part of every application. Even though we sometimes hate to write time and think they are just time consuming, they can make our app must more stable and bulletproof. One often scenario that we want to test is to check if some method in a class was called with certain parameters. We have a class Importer to read and import a record from XML.

We want to test if XML was correctly read and if our process(record) method was called with record object.

Solution

ArgumentCaptur enables us to capture actual object that is passed to our process(record) method. We don’t only test if method was called, but also if it was called with specific values.

Extra tip for collections

Again, let’s image we modify our method to be able to read multiple records.

Solution is similar. To test collection (List, Set, Map…) arguments, we have to use some additional Mockito features.

In our test, recordsCaptor.getValue() returns Set<Record>. We can check size or each record. So with Mockito you can test a lot of scenarios and it’s a really nice tool for unit testing.

OpenTSDB – the perfect database for your Internet of Things projects

- Last update:

I got a call the other day with a question: how can we store huge amount of sensor data. They are measuring air temperature in different rooms every 5 seconds. That means 17280 data points per data, 6307200 data points per year and for 15 rooms 94608000 data points per year.

Because I never had a situation where I needed to store a huge amount of sensor data, I didn’t know the answer. But I started digging. There are many questions online regarding what database to use to store this kind of data. Some recommend old-school databases like MySQL or Oracle. Some recommend Redis, Riak or MongoDB. But one recommendation beat them all: OpenTSDB.

OpenTSDB – The Scalable Time Series Database
Store and server massive amounts of time series data without losing granularity.

Currently in version 2.0, OpenTSDB is a tested solution build on top of HBase. It was designed especially for time series data and can handle

– up to 100+ billion data points and
– 2000 new data points per second (tested by OpenTSDB developers on a old dual-core Intel Xeon CPU from 2006; I tested on a newer machine and could easily insert 20000 points in few seconds).

Long story short. It’s perfect database for huge amount of sensor data. It has great options to query data (I will explain it below), has additional features to annotate data and it’s under active development.

Installation and running it for the first time

To run OpenTSDB, you need to first install HBase. The procedure is pretty straightforward. First you need to install HBase. Download HBase, unpack, define configuration and run it with

If everything was defined correctly, you should get a message

Next step is installing OpenTSDB. There is a great tutorial how to install OpenTSDB. In short, download it and unpack or clone git repository, and run build.

It should take few minutes to compile everything. Next step is to create tables with command

You can see created tables with few opensource HBase viewers like hrider. Currently the compression is set to none. It’s highly recommend to use compression LZO, because there is no performance impact but it can highly reduce the size of your data.

Because we will store temperatures in metric temperatures, we need to create it first. OpenTSDB has a configuration to enable auto creation of metrics, but it’s not recommended, so we will do it manually.

The last step is to run everything.

If everything went well, you should see OpenTSDB page at localhost:4242. It’s that simple.

How data is stored

How OpenTSDB is storing the data is in my opinion the biggest difference compared to other databases. It does support tables, but they are actually called metrics. In each metric we can store data points. Each data points is structured as

Timestamp (unix time or ISO 9601 format) is the time of the data point. Value is a number (integer or float). Then we have tags. With tags we separate data points. In our example, we are storing value for bedroom on our first floor. This structure enables us to separate data and later make advanced queries; for example average temperature on first floor or sum of all rooms.

Storing data

With version 2.0, OpenTSDB has 2 ways to store and access data (plus one additional to store by importing data). They are Telnet API, HTTP API and batch import from a file. Make sure you have OpenTSDB running before you try the examples below.

Storing with Telnet API [Java]

We need to execute command PUT with metric and data. še dopiši

Storing with HTTP API
When working with HTTP API, we have to make a PUT request to the URL localhost:4242/api/put with JSON data.

There is also a possibility to make a batch insert. Just wrap all metrics in an array.

Personally I had few problems inserting a large amount of data with the API. I ended up using Telnet API and it seems to work really well.

Querying the data

The whole beauty of OpenTSDB is it’s ability to not only to store huge amount of data, but to also query it fast. I will be showing how to query data with HTTP API, but the same query parameters can be used with Telnet API.

For the examples, we will first insert some data. Of course we can insert a much large dataset, but for this tutorial lets keep it simple.

Getting all temperatures

Let’s break down the request:
1. We can make GET or POST requests
2. The HTTP API URL is http://localhost:4242/api/query
3. We must define start, but end is optional. It can be unix timestamp or you can define nx-ago where n is unit and x is metric. For example, 1day-ago or 1h-ago. OpenTSDB will automatically convert it to timestamp based on your time.
4. m is the metric, where we are using aggregation = sum and metric = temperatures.
5. The last is grouping_operator (inside {}), which is used to group the data. If we define it with *, then it will not group the data. We can also use it to filter. For example room=bedroom will only fetch data from bedroom.

You can read more about different parameters and what they do at http://opentsdb.net/docs/build/html/api_http/query/index.html.

Our above request returns JSON.

Getting temperatures in the bedroom

As mentioned above, we can query by tags. In our case by room=bedroom.

returns

Getting average temperature on first floor

To calculate the average of the temperatures on first floor, we have to group by tags. Be careful to define correct aggregation function (in our case avg). See all aggregators at http://opentsdb.net/docs/build/html/user_guide/query/aggregators.html#available-aggregators.

produces

We can see tag room in aggregateTags. It means it used this tag to aggregate (or if you are familiar with other databases, GROUP BY) data.

Getting average temperatures per day

Let’s imagine a situation where we want to create reports of the temperatures on a daily basis. We could load all the data and then manually calculate the averages. For larger datasets it could take some time. OpenTSDB has an answer. We can also define downsampling. Downsampling will automatically calculate the values based on our downsampling aggregation function and timeframe.

Notice different parameter m? We added 1d-avg (be careful to separate everything correctly with “:”), which will downsample by 1 day and calculate average. Compared to manual way, it’s much faster and it just gives us results, which we can use in graphs.

Other awesome features

OpenTSDB has few additional features to cover real-life situations. Of course we can easily add more with plugins. But 2 of them worth mentioning are Annotations and CLI Tools.

Annotations
Annotations enable us to add additional meta data to data points. For example, we could store information when we opened and closed window in each room or when we changed the heating level.

Read more at http://opentsdb.net/docs/build/html/api_http/annotation.html.

CLI Tools

CLI Tools are just simple tools to perform additional task like fixing the data storage (in case if something breaks down), querying and deleting data and creating metrics. One of the most common tools I use it scan, because it has the feature to delete data. t’s useful when we are doing different tests.

To delete all temperatures for basement, we execute command

Again, we can filter what to delete with start and end parameters, metric and tags.

Wrap up

OpenTSDB has been proved to be an excellent solution. It’s scalable, fast and has really neat features. Most importantly, it’s under active development and has many people contributing. With the era of IoT and Big Data upon us, it has a bright future ahead.

If you are ready, start with http://opentsdb.net/docs/build/html/index.html.

Why I think Spring Data repositories are awesome – Part 1

- Last update:

I have been using Play Framework from version 1.2. Currently I’m using 2.2, which comes with a simple Ebean ORM but after reading few comments I realized it won’t be good enough for more complex projects. There is nothing worse than to realize in the middle of the project that some bug is causing your app not to work.

I looked around and noticed Spring Data. At first I didn’t put much effort in it but after checking the docs I realized it’s awesome solution. Only problem is that I doesn’t work out of the box with the Play Framework. There are few example projects on github, for me the only working was https://github.com/jamesward/play-java-spring.

I strongly advise you to check the code, because it’s show how to combine Play Framework with Spring Data. At the same time it shows how Dependency Injection works, how you define repositories and how to use them in controllers.

How it works

The basic logic is that Spring Data offers basic repositories for basic operations like saving, finding, deleting etc. Let’s imagine we have an entity Post.

I’m not going into details how to create an entity. There are many great tutorials around. For the entity we create a repository

1. We extended CrudRepository. CRUD stands for Create, Read, Update and Delete. It’s means that CrudRepository probably has some methods for creating, reading, updating and deleting posts. If we check the source of CrudRepository, it enables us to that.

2. We added annotation @Repository so Spring knows to find it correctly and inject it.

3. We extended CrudRepository where Post is out entity and Long is type of the primary key. Based on there 2 attributes, Spring knows how to correctly build queries.

Let’s take it for a spin

We put the example in 1 method for the sake of simplicity. Point is that we can simply retrieve, insert, update and delete Post entity. Spring handles building the queries, converting to objects and all other things. Again, check the example on github I mentined before so you know how to correctly define Controller, what @AutoWired does and why method index() is not static any more.

I have been using Spring Data for some time and there was not a single situation I couldn’t solve. I think the guys at Spring did a really good job. The simplicity on one side and ability to solve even most complex scenarios makes it really awesome.

More to come

In the next part we are going to check how to make a more complex queries by just defining methods name. We will also check how to include paging, sorting and how to run really really really complex queries.

google-http-java-client and java.lang.NumberFormatException: For input string: ” 0″ bug

- Last update:

I was developing a Java app to fetch a JSON data and process it. Because the the app will be running on GAE, I had to use GAE URL Fetch Java API. google-http-java-client library has this supported with UrlFetchTransport so it’s a clear signal to use it.

I wrote the simple script, tested it locally and uploaded to GAE. Tried it for the first time and it worked. Tried the second time, it failed. Tried few times more and it was failing at random interval. Full stack trace:

I had no idea what is going on. So I tried to investigate it:

1. Checked if I’m using latest versions -> Yes
2. Checked online if anyone else had same problem -> No
3. Tried to use same examples as in docs -> Same error
4. Made a request to some other servers -> It worked

The last one gave me a signal that something is wrong with the server I’m making requests to. So I made a test:

1. Make a request to some other server (which works)
2. Make request to primary server (which does not work)

I compared response headers. At first I didn’t notice it, but response from primary server was not returning Cache-Control header. Problem? I knew that primary server was using apache and mod_proxy. Main app that is returning JSON is behind a proxy server. So I had to update settings and updated the app which returns JSON to also send Cache-Control: private.

Private indicates that all or part of the response message is intended for a single user and MUST NOT be cached by a shared cache, such as a proxy server.

So the solution is actually pretty simple. Just make sure the Cache-Control: private is included in the response headers.

Running cron tasks with Django and Kronos

- Last update:

There are many ways to run scheduled jobs. More complicated solution can be developed with Celery, Advanced Python Scheduler, python-crontab or with many other solutions. But if you are looking for a quick and simple way to run scheduled jobs, then cron is the right tool (if you are using Windows, then services are similar alternative).

Running a simple python script using cron is simple. First we need to edit crontab

Every line defined one cron task. Every task has to have correct syntax

and our script.py

We will not go into details. If you want to read more about it, there are many articles. When we save, we are notified (if we correctly defined everything) that task has been installed. In our case, it will run our script.py every 2 hours.

Mixing it with Django

Running cron with Django can be a little bit more complicated. The reason lays in fact that Django needs settings to be included to run correctly. This can sometimes be pain in the ass because of the paths.

Easy way to solve this is to use Knosos.

Kronos makes it really easy to schedule tasks with cron.

With Knosos we need to register tasks.

What will this do? It will register a task and run is every 2 hours. It will automatically include everything and define correct path, so we can also use our app code; for example models.

To test everything, we run

and to register it

If we check crontab, we will see something similar to

If we did everything correctly, it should work and run our talk method every 2 hours.

Extra advice

While running tasks on my Linux machine, I was always getting email from Cron Deamon. This can be annoying, so we can simply “disable” it. Actually, we can redirect cron output to /dev/null. Just add setting KRONOS_POSTFIX.

Sending emails with mandrill.com, Djrill and Django

- Last update:

Last day I noticed a really cool solution for sending email from MailChimp called Mandrill. It offers free package up to 12000 emails per month and 250 emails per hour. I decided to include it in our project (Django) and give it a go.

Mandrill is a scalable and affordable email infrastructure service, with all the marketing-friendly analytics tools you’ve come to expect from MailChimp.

Because it’s built by famous MailChimp, it has to be really good. I found a nice library for Python called Djrill which nicely wraps django email sending capabilities.

1. Create an account on Mandrill and get API Key

Registration is pretty easy, just follow instructions at https://mandrill.com/signup/

2. Install Djrill (I pulled steps from official docs)

Install Djrill with pip:

Edit your project’s settings.py:

3. Start sending emails

The beauty of Djrill it you don’t have to change code, because it works with current code for sending emails. There are 2 ways to send an email

First:

Or second with EmailMultiAlternatives

There is also a cool feature to add tags and metadata to emails. Later in Mandrill you can filter emails by them. They can be useful to filter different types of emails (registration, newsletter) or for example, to whom you have sent an email (client a, client b, …)

psql: could not connect to server: No such file or directory Fix

- Last update:

Last day I was reinstalling postgresql on my Ubuntu 12.04 server. I started with normal commands

The installation completed successfully. I was also getting the locale warning message

and as I remembered from before, the fix for this is to add

at the bottom of file /etc/environment. All looked ok, but when I started database

I didn’t get any message. Then when I logged into postgres with

and wanted to create database, I got warning

I checked around and a lot of them mentioned that I had wrong or incomplete settings. Because I didn’t know what to do, I asked around (postgresql channel on freenode) and got really simple solution. I just had to run command

where 9.1 is version of my postgresql database. This recreated settings file and now everything works great.

How to test file uploads in Play Framework [Java]

- Last update:

Working on MEDinar, we are doing unit test for test our app. One of the scenario is when users upload their slides. We need to check if everything works. But the problem is I could not find a working example in Java how to test file upload. So after digging and reading few examples how to do it in Scala

https://groups.google.com/forum/#!msg/play-framework/UVi2kFxNoiE/VzQz4o8Hao4J
http://stackoverflow.com/questions/15133794/writing-a-test-case-for-file-uploads-in-play-2-1-and-scala

I developed a working solution for Java. We have a controller action

and code to test it

How to read local JSON file with Phonegap + jQuery Mobile

- Last update:

While working on my Phonegap project I wanted to load initial data into SQLite. One option is to download it from the internet, but it’s always annoying to notify the user to turn on internet and to wait for all the files to download (especially if you have to download 50 images).

The other option is to load local data. Problem is that identifying the right path to local file can be tricky. But the solution is actually easy.

All we need is to get current location, remove index.html to get root folder, append it to path of JSON file and voila.

How to get random element from array with JavaScript

- Last update:

Imagine you have an array of numbers.

Goal is to get a random element of an array. There are several solutions posted on stackoverflow.com. But most of them are just too complicated, so I created my solution.

I have extend Array to add function random. What is does it uses random number and length of array to create a random index. Let’s use it in action.

That’s it. Happy coding.

Update:

Actually it’s very bad practice to extend prototype. Reasons are problems with browser compatibility. It’s better to create a function and use it. Or you can always use libs like underscore and it’s method sample.