Monday, April 18, 2016

When a MYSql Stored Procedure Error isn't: Syntax Error on Line 1

This one cost me three hours today.

Easiest stored procedure in the world: originally much more complex, but I base cased it.

CREATE DEFINER=`root`@`localhost` PROCEDURE `myStoPro`(
IN p_keyval VARCHAR(45)
INSERT INTO touchhistory (
    VALUES (

No good. Check your SQL Syntax at line 1 near ')'.

Oh did all the SQL guys got on about this one. You are defining the query incorrectly. You have to do this and that, etc. etc. None of it made sense, the basic documentation clearly shows this is a valid stored proc and it validated in two MySQL tools. Regardless of "better" ways, this way should work and work fine.

So, I removed the query from the DB entirely. And STILL got the same error.

So I knew it had nothing to do with the query at all, and went to the Python code.

It turns out that you can receive this very misleading error if you are using the Python MySQLdb interface. If any of your arguments are previously unset/undefined variables, it will throw the Stored Procedure Syntax error.

for instance, say you do this:

cursor = db.cursor()
args = [keyval] # note that keyval has not been set, previously declared, etc
cursor.callproc ('myStoPro', args)

You will get the MySQL stored procedure syntax error.

Sunday, January 10, 2016

Angular: Testing (Karma/Jasmine) "controllerAs" Controllers that use External Templates and ng-repeat

I've been working with Angular professionally for quite a while now, hours and hours every day. Over time, you learn a lot about what's good and bad, and adjust. So, some of the ways I do things are not what you'd typically expect based on the usual books and Stack Overflow answers. For instance, I never use link functions...ever...and everything is a module. The only things those modules contain are controllers, directives, and services (and I do use .constant, that's handy).

This thinking tracks with Angular 2.0, where they have done away with all the variety of services and so on (everything is a component). So be prepared to let that all go.

That said, this article is an attempt to pull together many things that I have been asked and sought answers to in relation to basic project structure and testing. Those things include:
  • Should I use JQuery, or is JQLite enough?
  • How do I do basic tests of a controller if I use ngRoute.
  • How do I test directives with "controllerAs" syntax.
  • How do I access the controller of a directive compiled with controllerAs syntax.
  • How do I test a directive contained in a template of another directive is working. 
  • How do I trigger an ng-repeat in a compiled directive, and test that it's working.

The project and code are deliberately bare bones, focusing on the basic code. Robustness is absent to minimize the code you need to understand.

Here's the GitHub link. I will reference that as opposed to repeating the code in this article.


You have node and the latest Chrome installed (you can add Firefox if you like, but I use Chrome), are familiar with the basic use of npm and running a simple node script.

View the README for more detail, including what commands fire up the app and to run the tests. If you want a list of the deps, open up package.json.

You should also be familiar with the basic setup and use of Karma and Jasmine; this article isn't a tutorial on that tech. If you don't know the basics of how to use these technologies and you're an Angular developer (or really any kind of JS developer), you should stop reading this article right now and head over to the Karma site. Return when you've done a basic tutorial.


I use it for the powerful selectors in my directive template testing. JQLite's .find ( ) only works with tag names; if it worked like regular JQuery, I wouldn't use JQuery, but JQLite's .find ( ) is so hamstrung as to be kinda useless. Yes, you can work with and around the limitations of JQLite, and you can always use native selectors via $document and so on (and you may have to if JQuery is verboten). I just prefer not to. As long as you make sure you don't slip into "the JQuery way" in your Angular code, you're good.

Karma and Jasmine

Take a look at karma.conf.js. It's almost entirely the default generated by Karma. The one thing to really note if you're not familiar with it is the last "files" entry, and the preprocessor.

'**/*.html' means, "grab html files in nested directories off the base path". Since that's where all my template files for directives are, that's what I want. If I needed to exclude a file I could always add it to exclusions, and there are other options for this kind of "find all" thing, but this is all I need for my sample project.

Take a look at that preprocessor section. You see that I've pointed the preprocessor 'ng-html2js' at the '**/*.html' files. This tells Karma where to find my html template files (nested directories). The preprocessor is a plugin and does not come by default with Karma; take a look at the package.json and you'll see it's an explicitly installed dependency. The 'ng' should tell you that this is an Angular thing. There's another version available for non-Angular projects. I recommend you at least read some of the basics on this preprocessor, it's extremely useful.

Basic tests of a controller (mainview-controller.js, mainview-tests.js)

Pretty straightforward. If you look in app.js, you see the usual .config ngRouter setup; note the "controllerAs" syntax here. MainView and its controller are also pretty simple. Note the dependencies injected into the MainView module (at the bottom of the file). Sure, you could do all this upfront in app.js, and in some projects that's fine (all dependencies just hang off off the main app module), but that's contrary to the notion of everything being an independent module.

In mainview-tests.js, you see what you expect with one possible exception. You inject the module, get the controller using $controller ( 'NameOfController' ), feed it  the required $scope data, and proceed happily testing away. The exception is that 'NameOfController' is not the ngRoute 'controllerAs'. It makes sense if you think about it, since in your test you aren't using a page load via ngRoute to instantiate the controller.  Otherwise this should all be business as usual.

Testing a Directive that defines its controller using ControllerAs

There are two directives in the project; parent-directive.js, and child-directive.js. In the template for MainView, you'll find markup for a parent-directive. In parent-directive's template, you'll find markup, within an ng-repeat, for child-directive.

First, think though the flow; ngRoute loads MainController and it's corresponding view (mainview.html). That view contains a parent-directive, and feeds it an array as the "data" arg; notice that the array is accessed using the 'controllerAs' name defined in the ngRoute config section of app.js.

Parent-directive in turn compiles and runs the required number of child-directive(s), feeding them items using ng-repeat. Nothing fancy here.

We've already tested that the data for parent-directive exists and can be found on MainView's controller. Now, we want to test that when we compile and run parent-directive with some sample data, we can do so easily enough. Sure, we could always create some dummy data. But why not use the data that is actually on MainView's controller?

parent-directive is nothing unusual; you define the directive, define the props you want on the scope (isolates scope), point at the controller, define the controllerAs, and off you go. No link function, no nada, you don't need it. Just work with the $scope injected into the controller, and leave the scope alone otherwise; all handlers and whatnot go on the controller, and that's where you access them.

The most important thing to notice in parent-directive is how the html templates are made available, and how controllers are accessed.

I won't go into a long explanation on the html templates. I've seen many different ways to do this and of course every Angular developer will tell you their way is the "proper" way (and get angry when you even suggest it might not be). So I'll just say, this is how I do it, it's supremely testable, and nobody I work with has ever rejected it (in fact I get a lot of, "wow where'd you learn this," the answer being, "in Hell").

First note that we inject whatever html templates will be required to run this directive module. This includes the templates used by child directives. If you leave them out, when parent goes to compile child, you'll see a GET failure in the console.

Then we have to load the $templateCache. It might be worth some reading on your part if you've never gotten the gist of what $templateCache does. For now just know that even though we configured Karma to see our templates (the preprocessing stuff mentioned before), when we compile our directive in a test, it (the directive) will attempt to load the required template as usual (causing an error), unless (evidently) it is populated in the $templateCache. If it is, the text won't actually try to load the page and the test will complete.

Note that here I only load up the $templateCache entry for the directive we are compiling (parent-directive)...not the child-directive's template.

With that done, we move onto setting up the scope and accessing the controllers.

To grab a simple controller, like MainView, you just use $controller, the name of the controller (remember, not the "controllerAs" name), feed it the scope, and you're done. Now I can grab the array on MainView's controller and feed it as the "data" arg on the scope of the directive instead of using something dummy (nothing wrong with that, just illustrating the point here).

Grabbing the controller off of a directive using an external template and controllerAs is different. Here's the flow:
  • Set a reference to $rootScope so I can use it throughout the test suite if needed. You don't have to use $new ( ). 
  • Grab the MainView's controller the same way you did in mainview-tests.js. 
  • Set whatever data you want from the controller (or just dummy stuff) onto the scope.
  • Set up your directive template. Notice that the "data" arg feeds in the data by whatever you named it on the scope (which should match the scope spec in the directive, for instance, scope : { data : '=' }. The fact that both the attribute and the val are both "data" is just a result of my convention of using "data", you can call it whatever you like.
  • Compile the directive. Because we set up the $templateCache, it'll find it's template.
  • Apply the scope ( don't need to use $digest. $apply works fine and is, as far as I know, the recommended mechanism). This will, for example, make the directive use the data and generate the items for ng-repeat. I put this in bold because it can be VERY confusing to find out how to do this if you just search for it. This, for me, is the most testable and terse way to get the job done.  
  • Grab the directive's controller by using either scope ( ) if the scope isn't isolated, or isolateScope if it is. A catch all is "dirt.scope ( ) || dirt.isolateScope ( )". Again, note that this is different than getting a simple view controller. 

The child-directive test is more or less the same as the parent-directive. To test the data in child-directive, I use a dummy object. I could have instantiated a MainView controller again like you've seen previously, but wanted to illustrate the use of dummy data.  

So that's it. This is most of the baseline structure I use for all my pre-2.0 Angular work these days. I tend to find it much simpler to work with, and more straightforwardly testable, than other seeds I've found.

As always, thanks for visiting. 

Tuesday, October 6, 2015

XBox IE and Angular dynamic styles

Some errata for you:

I was testing an application across some tablets, laptops and phones, and found some of the usual required UI tweaks to make bootstrap a little more cooperative and so on, but this one I just ran into and it seems particular to IE on XBox, which I tested just for the heck of it.

I had coded up the styles inline to get it right before moving it to CSS; this was just me working through it quickly, so I know that the second way here is more correct, and that switching
on the ng-class is typically preferred.

If you set a style dynamically this way, it won't work (only on XBox IE):

<div style="padding-top: 1px; background-color: {{ colors [ data.colorname ] }}">

If you set it this way, it will

<div ng-style="{ 'padding-top' : '1px', 'background-color' : colors [ data.colorname ] }">

The peculiarity is, whether or not it should be done this way, the first one works everywhere, on every phone, tablet, and laptop I had available to check it on (two macs, two PCs, two phones), but the background color did not render on XBox IE.

I changed to the second form and it worked fine.

Again, just some errata.

As always, thanks for visiting.

Thursday, October 1, 2015

Acronyms in Angular Directive names

Here's an interesting quick one.

Many industries live and breathe acronyms. So, it would be no surprise if they wanted to use them in directive names.

Say you were creating a management system. PM is a pretty standard acronym for Project Manager.

If you name your directive this:


Then what you might expect, with Angular doing its funky "you wrote it this way but we'll divide it up with dashes this way" thing:


Nope. You will have to use it like this as a directive (assuming attribute restriction in the directive itself):


Or, rename the directive:


Angular evidently does not recognize all-caps acronyms. Certainly not a deal breaker, but I'd like to think that if I was building out that mechanism within Angular I'd have thought of this one.

If there is a solution to this that is very straightforward and uses the framework as-is, nobody I asked is aware of it.

Yes...I am aware of the documentation and spec for this. My point is that a little case detection in the mechanism could solve it. Perhaps a good interview question.

There is also no debug error of course. One of the frustrating things about using directives is, any typo and it just won't work, but nothing will tell you why. A test can fail, but it won't tell you that you declared the markup incorrectly or that the directive didn't start up (why would it). It's one of those sort of old school "you just have to know" things. Ultimately you could probably write a test to figure that out but...well, I don't have all day to toy around with creative tests.

As always, thanks for visiting.

Thursday, May 28, 2015

Getting an Entity By ID: Google Datastore (using Scala)

Here's another one of those, "it should be so simple" ones.

If you store an Entity in the Google Datastore, and you want to retrieve it by the auto-generated "Long" ID, none of the things you think would

SELECT * FROM mytype WHERE id = 12345, and so

Note that the auto-generated column where the ID is put, is actually shown as "ID/Name" (but using that in any form doesn't help either).

You have to work with a Key (the below code assumes you have already created a new Query instance):

Note that I make sure I convert myVal to a Long (these two lines of code are actually in a "match", which via map iteration checks if the "v" in ( k, v ) is a Double, so I'm guarded against the exception you may have balked at), because when I send over the JSON from the UI, it appears as a Double on the server side, which won't work (and I tried everything short of getting into custom serialization).

Anyway, I banged around a couple of hours on this one, maybe it saves somebody some grief.

As always, thanks for visiting.  

Thursday, May 21, 2015

Scala: Get Google DataStore PreparedQuery Entity list, convert to JSON Array

This was an interesting one. Using Scala, I had to do something pretty simple with some stored Entities in the Google DataStore:

val query : Query = new Query ( "mytype" )
val pq : PreparedQuery = datastore.prepare ( query )

"pq" now contains an iterable list (either as an iterator or a java list), of Entity objects that are "mytype". Great, the datastore is really easy to work with...but I need JSON.

If you just "toString" these lists, you get a lot of "->" symbols and such, that of course aren't JSON. But without excessive iteration, matching, implicit conversions, and so forth, how can I just take that list in "pq" and somehow transform it into JSON?

It turned out there's a great utility, JSONArray ( val : List ). It will turn a Scala list into a nicely formatted JSON Array. Note that it will NOT transform the entities, you have to do that separately (as you'll see, fortunately it's simple).

The tricky thing was, that the List returned from the PreparedQuery isn't compatible with that JSONArray constructor, because it's a Java list (as opposed to a Scala one).

So, for the heck of it, I tried this, and it compiled...but didn't run.

new JSONArray ( pq.asList ( withLimit ( 1000 ) )
.asInstanceOf[List[Entity]] )

The error was that a "LazyList of type java. (etc. etc.) can't be cast to immutable Scala list". Including JavaConversions and so on didn't work. At runtime, the code would break.

I figured there is probably some way around this if I kept researching and experimenting; I went through my Scala book, Stack Overflow and so on, and found/came up with a variety of different ways of mapping and moving around the entities in the lists. Some of it even ran, but nothing seemed stable under all conditions unless I wrote a lot of edge case code. I was moving away from a "keep it simple" answer which I could easily rework if I found a clearly better solution.

So I came up with this, and it worked for everything I threw at it. Iterate the Java list using the asIterator returned by the PreparedQuery, pass each item to a new JSONObject, and add that JSONObject into a Scala ListBuffer (a very nice "in progress" list utility). Then get the ListBuffer.toList and pass it to JSONArray...viola, all entities (and their properties) in the PreparedQuery entity list are now a JSONLint worthy JSON array.

Note that in my Entities, I store either simple values (strings numbers etc.) or valid JSON objects. So I don't have to worry about nested collections; if you do, you might have to do some recursion and matching/transforming here. Not rocket science but a higher order of complexity...I recommend just storing JSON or basic values if you can.

Notice I use a ListBuffer, which creates an "in progress" list that you can append (as opposed to an immutable Scala List or fixed dim Array). Later, ListBuffer is submitted to JSONArray as a plain old Scala List.

Anyways, here's the code:

When I catch the response.write of this in my JavaScript, all the JSON is fine. As always, thanks for visiting. 

Monday, May 4, 2015

How To: Use JavaScript/JSON object in Scala Servlet

I'm relatively new to Scala. I really like working with it, far more so than Java. Straightforward callback functions, and processing instructions fed into course-grained iterators and so on (which tends to lead toward higher-order factoring instead of lower-order sub typing), are really a better way to program imho. Not to mention traits, mixins, and other such nerd fun.

It's not always easy though. One such area is JSON.

JSON isn't supposed to be ponderous to work with. I should be able to do something like this...

> Get JSON string
> Turn it into JSON Object (or from object into string)
> Easily access a key->val.
> Turn the whole thing into a string

...with a couple of lines of code.

I've looked through a number of approaches. The examples I've found using the "native" Scala JSON utils tend to be ponderous (write a wrapper class that goes through opts and matches against...), or, you end up chasing down a third party library. I have no issue with third party libs, but I do believe that if the language/environment does something important natively, you should know how to use the basics of it, and I am indeed trying to do something very basic.

Anyway, I worked through java immutable map to scala immutable map conversions and everything else people are trying to work around, and landed on this amazingly simply routine. Note that I couldn't say for sure if this is the best way under all conditions, but it's working flawlessly for me (I'm building an app on the Google App Engine using Scala).

I'm sending a very basic JSON object from my JS front end using, it has two keys, and one of the vals is a nested object (which can of course have other nested objects etc. as long as it's all valid JSON):

{ "dstype":"registration","dsdata":{"val1":0.5,"val2":"Hello"}}

I get it in the servlet's doPost method, and access like this. Note that yes, I'm aware that the req.getParameterNames.nextElement isn't semantically pleasing. But it returns exactly what you want, which is that incoming JSON data. And I've omitted exception handling for brevity:

// The objects I'm using, sure you could just ._ but let's get specific here
import scala.util.parsing.json.JSONObject
import scala.util.parsing.json.JSON

// gets the JSON string. I know it looks a little odd,
// and there's hopefully a more semantically and equally brief way to do this,
// but this returns the JSON as posted by the front end, and that's what I need.
val jsonString = req.getParameterNames.nextElement.toString 

// parses it into object Some(Map( k -> v, k -> v,  ... ) )
val jsonObject = JSON.parseFull ( jsonString

// gets the value of "Some" from the jsonObject, which is the map you want
// note that this can throw a propertyNotFound exception if there is no value
val jsonVal = jsonObject.get 
// matches against type to run desired typed arguments,
// for example, here execute takes a String Map Any,
// or if you don't like infix notation, Map[String, Any],
// We can't just pass in jsonVal, there's not enough type info.
// Note that you could also use jsonObject.get.instanceOf[String Map Any],
// but as I understand it, casting is frowned upon in Scala in favor of matching.
// I never liked the Java way of casting around map, which forced you to put 
// in the RepressWarnings attribute, so I'm going with the flow. 

jsonVal match {
    case m:Map[String,Any] => new MyCommand ( ).execute ( m )

That's it, execute now gets a String Map Any (or Map[String, Any] if you prefer), which you deal with as a usual Scala map. 

Note that what I actually do, is pass a callback to execute as a partially applied function. Partially applied functions are very cool.

case m:Map[String,Any] => new MyCommand ( ).execute ( m, onMyCallback _ )

execute looks like this:

 def execute ( params : String Map Any, 
               callback : (String Map Any) => Unit ) = {}

In the command's execute class, when all is done, I pass the result to the callback, which executes and writes the response to the client. Note here (remember I'm back in the servlet now via the callback), I just use a new scala JSONObject ( response ).toString( ) to serialize the data. 

def onMyCallback ( data : String Map Any ) = {
      var response = Map[ String, Any ] ( )
      response += ( "status" -> "ok" )
      response += ( "data" -> "MyServlet - Command executed" )
      resp.getWriter.print ( new JSONObject ( response ).toString() )

Anyways, all this finally let me stop noodling with Scala json code and get down to the business of finishing up my project, which I have to do somewhat quickly so I can get back to the paying work.

As always, thanks for visiting.