Thursday, May 21, 2015

Scala: Get Google DataStore PreparedQuery Entity list, convert to JSON Array

This was an interesting one. Using Scala, I had to do something pretty simple with some stored Entities in the Google DataStore:

val query : Query = new Query ( "mytype" )
val pq : PreparedQuery = datastore.prepare ( query )


"pq" now contains an iterable list (either as an iterator or a java list), of Entity objects that are "mytype". Great, the datastore is really easy to work with...but I need JSON.

If you just "toString" these lists, you get a lot of "->" symbols and such, that of course aren't JSON. But without excessive iteration, matching, implicit conversions, and so forth, how can I just take that list in "pq" and somehow transform it into JSON?

It turned out there's a great utility, JSONArray ( val : List ). It will turn a Scala list into a nicely formatted JSON Array. Note that it will NOT transform the entities, you have to do that separately (as you'll see, fortunately it's simple).

The tricky thing was, that the List returned from the PreparedQuery isn't compatible with that JSONArray constructor, because it's a Java list (as opposed to a Scala one).

So, for the heck of it, I tried this, and it compiled...but didn't run.

new JSONArray ( pq.asList ( withLimit ( 1000 ) )
.asInstanceOf[List[Entity]] )


The error was that a "LazyList of type java. (etc. etc.) can't be cast to immutable Scala list". Including JavaConversions and so on didn't work. At runtime, the code would break.

I figured there is probably some way around this if I kept researching and experimenting; I went through my Scala book, Stack Overflow and so on, and found/came up with a variety of different ways of mapping and moving around the entities in the lists. Some of it even ran, but nothing seemed stable under all conditions unless I wrote a lot of edge case code. I was moving away from a "keep it simple" answer which I could easily rework if I found a clearly better solution.

So I came up with this, and it worked for everything I threw at it. Iterate the Java list using the asIterator returned by the PreparedQuery, pass each item to a new JSONObject, and add that JSONObject into a Scala ListBuffer (a very nice "in progress" list utility). Then get the ListBuffer.toList and pass it to JSONArray...viola, all entities (and their properties) in the PreparedQuery entity list are now a JSONLint worthy JSON array.

Note that in my Entities, I store either simple values (strings numbers etc.) or valid JSON objects. So I don't have to worry about nested collections; if you do, you might have to do some recursion and matching/transforming here. Not rocket science but a higher order of complexity...I recommend just storing JSON or basic values if you can.

Notice I use a ListBuffer, which creates an "in progress" list that you can append (as opposed to an immutable Scala List or fixed dim Array). Later, ListBuffer is submitted to JSONArray as a plain old Scala List.

Anyways, here's the code:


When I catch the response.write of this in my JavaScript, all the JSON is fine. As always, thanks for visiting. 

No comments:

Post a Comment