Thursday, May 28, 2015

Getting an Entity By ID: Google Datastore (using Scala)

Here's another one of those, "it should be so simple" ones.

If you store an Entity in the Google Datastore, and you want to retrieve it by the auto-generated "Long" ID, none of the things you think would work...work.

SELECT * FROM mytype WHERE id = 12345, and so on...fail.

Note that the auto-generated column where the ID is put, is actually shown as "ID/Name" (but using that in any form doesn't help either).

You have to work with a Key (the below code assumes you have already created a new Query instance):



Note that I make sure I convert myVal to a Long (these two lines of code are actually in a "match", which via map iteration checks if the "v" in ( k, v ) is a Double, so I'm guarded against the exception you may have balked at), because when I send over the JSON from the UI, it appears as a Double on the server side, which won't work (and I tried everything short of getting into custom serialization).

Anyway, I banged around a couple of hours on this one, maybe it saves somebody some grief.

As always, thanks for visiting.  

Thursday, May 21, 2015

Scala: Get Google DataStore PreparedQuery Entity list, convert to JSON Array

This was an interesting one. Using Scala, I had to do something pretty simple with some stored Entities in the Google DataStore:

val query : Query = new Query ( "mytype" )
val pq : PreparedQuery = datastore.prepare ( query )


"pq" now contains an iterable list (either as an iterator or a java list), of Entity objects that are "mytype". Great, the datastore is really easy to work with...but I need JSON.

If you just "toString" these lists, you get a lot of "->" symbols and such, that of course aren't JSON. But without excessive iteration, matching, implicit conversions, and so forth, how can I just take that list in "pq" and somehow transform it into JSON?

It turned out there's a great utility, JSONArray ( val : List ). It will turn a Scala list into a nicely formatted JSON Array. Note that it will NOT transform the entities, you have to do that separately (as you'll see, fortunately it's simple).

The tricky thing was, that the List returned from the PreparedQuery isn't compatible with that JSONArray constructor, because it's a Java list (as opposed to a Scala one).

So, for the heck of it, I tried this, and it compiled...but didn't run.

new JSONArray ( pq.asList ( withLimit ( 1000 ) )
.asInstanceOf[List[Entity]] )


The error was that a "LazyList of type java. (etc. etc.) can't be cast to immutable Scala list". Including JavaConversions and so on didn't work. At runtime, the code would break.

I figured there is probably some way around this if I kept researching and experimenting; I went through my Scala book, Stack Overflow and so on, and found/came up with a variety of different ways of mapping and moving around the entities in the lists. Some of it even ran, but nothing seemed stable under all conditions unless I wrote a lot of edge case code. I was moving away from a "keep it simple" answer which I could easily rework if I found a clearly better solution.

So I came up with this, and it worked for everything I threw at it. Iterate the Java list using the asIterator returned by the PreparedQuery, pass each item to a new JSONObject, and add that JSONObject into a Scala ListBuffer (a very nice "in progress" list utility). Then get the ListBuffer.toList and pass it to JSONArray...viola, all entities (and their properties) in the PreparedQuery entity list are now a JSONLint worthy JSON array.

Note that in my Entities, I store either simple values (strings numbers etc.) or valid JSON objects. So I don't have to worry about nested collections; if you do, you might have to do some recursion and matching/transforming here. Not rocket science but a higher order of complexity...I recommend just storing JSON or basic values if you can.

Notice I use a ListBuffer, which creates an "in progress" list that you can append (as opposed to an immutable Scala List or fixed dim Array). Later, ListBuffer is submitted to JSONArray as a plain old Scala List.

Anyways, here's the code:


When I catch the response.write of this in my JavaScript, all the JSON is fine. As always, thanks for visiting. 

Monday, May 4, 2015

How To: Use JavaScript/JSON object in Scala Servlet

I'm relatively new to Scala. I really like working with it, far more so than Java. Straightforward callback functions, and processing instructions fed into course-grained iterators and so on (which tends to lead toward higher-order factoring instead of lower-order sub typing), are really a better way to program imho. Not to mention traits, mixins, and other such nerd fun.

It's not always easy though. One such area is JSON.

JSON isn't supposed to be ponderous to work with. I should be able to do something like this...

> Get JSON string
> Turn it into JSON Object (or from object into string)
> Easily access a key->val.
> Turn the whole thing into a string

...with a couple of lines of code.

I've looked through a number of approaches. The examples I've found using the "native" Scala JSON utils tend to be ponderous (write a wrapper class that goes through opts and matches against...), or, you end up chasing down a third party library. I have no issue with third party libs, but I do believe that if the language/environment does something important natively, you should know how to use the basics of it, and I am indeed trying to do something very basic.

Anyway, I worked through java immutable map to scala immutable map conversions and everything else people are trying to work around, and landed on this amazingly simply routine. Note that I couldn't say for sure if this is the best way under all conditions, but it's working flawlessly for me (I'm building an app on the Google App Engine using Scala).

I'm sending a very basic JSON object from my JS front end using JQuery.post, it has two keys, and one of the vals is a nested object (which can of course have other nested objects etc. as long as it's all valid JSON):

{ "dstype":"registration","dsdata":{"val1":0.5,"val2":"Hello"}}

I get it in the servlet's doPost method, and access like this. Note that yes, I'm aware that the req.getParameterNames.nextElement isn't semantically pleasing. But it returns exactly what you want, which is that incoming JSON data. And I've omitted exception handling for brevity:

// The objects I'm using, sure you could just ._ but let's get specific here
import scala.util.parsing.json.JSONObject
import scala.util.parsing.json.JSON

// gets the JSON string. I know it looks a little odd,
// and there's hopefully a more semantically and equally brief way to do this,
// but this returns the JSON as posted by the front end, and that's what I need.
val jsonString = req.getParameterNames.nextElement.toString 

// parses it into object Some(Map( k -> v, k -> v,  ... ) )
val jsonObject = JSON.parseFull ( jsonString

// gets the value of "Some" from the jsonObject, which is the map you want
// note that this can throw a propertyNotFound exception if there is no value
val jsonVal = jsonObject.get 
    
// matches against type to run desired typed arguments,
// for example, here execute takes a String Map Any,
// or if you don't like infix notation, Map[String, Any],
// We can't just pass in jsonVal, there's not enough type info.
// Note that you could also use jsonObject.get.instanceOf[String Map Any],
// but as I understand it, casting is frowned upon in Scala in favor of matching.
// I never liked the Java way of casting around map, which forced you to put 
// in the RepressWarnings attribute, so I'm going with the flow. 

jsonVal match {
    case m:Map[String,Any] => new MyCommand ( ).execute ( m )
}

That's it, execute now gets a String Map Any (or Map[String, Any] if you prefer), which you deal with as a usual Scala map. 

Note that what I actually do, is pass a callback to execute as a partially applied function. Partially applied functions are very cool.

case m:Map[String,Any] => new MyCommand ( ).execute ( m, onMyCallback _ )

execute looks like this:

 def execute ( params : String Map Any, 
               callback : (String Map Any) => Unit ) = {}

In the command's execute class, when all is done, I pass the result to the callback, which executes and writes the response to the client. Note here (remember I'm back in the servlet now via the callback), I just use a new scala JSONObject ( response ).toString( ) to serialize the data. 

def onMyCallback ( data : String Map Any ) = {
      var response = Map[ String, Any ] ( )
      response += ( "status" -> "ok" )
      response += ( "data" -> "MyServlet - Command executed" )
      resp.getWriter.print ( new JSONObject ( response ).toString() )
}

Anyways, all this finally let me stop noodling with Scala json code and get down to the business of finishing up my project, which I have to do somewhat quickly so I can get back to the paying work.

As always, thanks for visiting.