JSON support has been introduced in Delphi 2010 as a part of DBExpress database driver architecture, but of course JSON support is not limited to just database applications. JSON is similar to XML as both are text-based data interchange formats. Delphi 6 introduced “TXMLDocument” component and the “XML Data Binding Wizard” to make it easier to work with XML documents in code. In this paper I’m presenting a similar component for JSON called “TJSONDocument”. The next step was to implement “TJSONTreeView” component for displaying the content of a ”TJSONDocument” in VCL Forms applications. Using these components a simple JSON Viewer application has been created and described here. In the XML world there are two categories of parsers: DOM and SAX. DOM parsers takes XML string and builds its in-memory representation – a “Document Object Model” - for application to traverse and update it. At the other hand SAX is a streaming interface — applications receive information from XML documents in a continuous stream, with no backtracking or navigation allowed. In the later part of this article I’m describing the “TJSONParser” component that was created as an experimental TJSONDocument-descendant that provides SAX-like event-based processing for JSON. The full source code described in this paper can be downloaded from [1, 2] https://radstudiodemos.svn.sourceforge.net/svnroot/radstudiodemo s/branches/RadStudio_XE/Delphi/DataSnap/JSONViewer See “References” section at the end of this article.
Why JSON? JSON is relatively new as it was first described by Douglas Crockford in July 2006 in his IETF Request for Comments “The application/json Media Type for JavaScript Object Notation”[2]. In many respects JSON is similar to XML as both are text based data interchange formats used broadly in the Web. While XML has now became the whole family of related standards – including XML Namespaces, XML Schema, XSL, XPath and others – JSON defines only a small set of formatting rules for the portable representation of structured data. The key strength of JSON is simplicity. Douglas Crockford describes JSON structure in his paper presented at XML 2006 Conference in Boston “JSON: The Fat-Free Alternative to XML” [3]: The types represented in JSON are strings, numbers, booleans, object, arrays, and null. JSON syntax is nicely expressed in railroad diagrams. Hide
image
JSON only has three simple types – strings, numbers and Booleans – and two complex types – arrays and objects. A string is a sequence of zero or more characters wrapped in quotes with backslash escapement, the same notation used in most programming languages. A number can be represented as integer, real, or floating point. JSON does not support octal or hex. It does not have values for NaN or Infinity. Numbers are not quoted. A JSON object is an unordered collection of key/value pairs. The keys are strings and the values are any of the JSON types. A colon separates the keys from the values, and comma separates the pairs. The whole thing is wrapped in curly braces. The JSON array is an ordered collection of values separated by commas and enclosed in square brackets. The character encoding of JSON text is always Unicode. UTF-8 is the only encoding that makes sense on the wire, but UTF-16 and UTF-32 are also permitted. JSON has no version number. No revisions to the JSON grammar are anticipated. JSON has become the X in Ajax. It is now the preferred data format for Ajax applications. The most common way to use JSON is with XMLHttpRequest. Once a response text is obtained, it can quickly be converted into a JavaScript data structure and consumed by a program JSON's syntax is significantly simpler than XML, so parsing is more efficient. JSON doesn't have namespaces. Every object is a namespace: its set of keys is independent of all other objects, even exclusive of nesting. JSON uses context to avoid ambiguity, just as programming languages do. JSON has no validator. Being well-formed and valid is not the same as being correct and relevant. Ultimately, every application is responsible for validating its inputs. Below is a fragment of sample JSON text, based on “Sample Confabulator Widget” from [4], used
in the later part of this article. { "widget": { "debug": "on", "window": { "title": "Sample Konfabulator Widget", "name": "main_window", "width": 500, "height": 500 }, "misc": ["hello", 23, false] } }
Delphi and DBXJSON.pas There is a big number of programming languages that have built-in support for JSON or libraries to work with JSON. These JSON bindings for different programming languages are listed on JSON homepage [4], including three open source Delphi libraries. Since Delphi 2010 the JSON support is part of the VCL library as implemented in the DBXJSON.pas unit. In order to visualize Delphi classes responsible for JSON support, I have added the “DBXJSON” unit directly to a little test Delphi application, clicked on “Model Support” tab in Project Manager and got the following UML class diagram. Some of the classes from DBXJSON.pas unit not related directly to JSON support are not shown here. Hide
image
In Delphi 2007 the DBX database driver framework architecture has been reengineered in pure Delphi code and introduced a number of interesting features including extensible command types. In the next release – Delphi 2009 – the DBX architecture has been extended and DataSnap framework for building client/server and multi-tier application has been reengineered as well as the extension of the new DBX architecture. One of the most interesting and powerful capabilities introduced to DataSnap support in the following Delphi release – Delphi 2010 – were lightweight callbacks passed to DataSnap server methods for the server application to be able to call back into
the client. The DBXJSON.pas unit defines the abstract base class for callback objects that contains “Execute” method that accepts and returns parameters of TJSONValue type, thus making it possible to pass in and out arbitrarily complex data structures encoded as JSON. Here is the declaration of this class: TDBXCallback = class abstract public function Execute(const Arg: TJSONValue): TJSONValue; virtual; abstract; // … other members stripped out end; The DBXJSON unit contains also functionality to parse JSON text into the graph of TJSONValuedescedants and to generate JSON text from the graph of objects in memory. On the graphics “TJSONAncestor.Owned: Boolean” property has been expanded to underline the fact that all JSON descendants have the “Owned” property that controls the lifetime of JSON objects in memory. TJSONObject class contains static method “ParseJSONValue” that effectively implements JSON parser functionality. It accepts a string parameter with JSON text and returns a TJSONValue reference to the root of the graph of TJSONAncestor-descendants. It is also possible to generate JSON text from the in-memory tree of JSON objects calling “ToString” overloaded method on any of “TJSONAncestor” descendants.
TJSONDocument Component Before Delphi 2010 introduced DBXJSON unit, I was trying to implement JSON parsing functionality manually coding JSON railroad diagrams. With DBXJSON implementation in place there is little point in reinventing the wheel; however there is still no design-time support for JSON. Everything has to be done in code. Hence the concept of creating a minimal VCL component
wrapper
for
JSON
parser
implementation
provided
by
“TJSONObject.ParseJSONValue” class method that accepts JSON text and returns the object tree representing JSON document structure in memory. “TJSONDocument” component has been implemented inside “jsondoc” unit to keep some level of parity with its “TXMLDocument” component equivalent that is implemented inside “xmldoc” unit. Below is the declaration of the “TJSONDocument” VCL component: unit jsondoc; //… type TJSONDocument = class(TComponent)
private FRootValue: TJSONValue; FJsonText: string; FOnChange: TNotifyEvent; procedure SetJsonText(const Value: string); procedure SetRootValue(const Value: TJSONValue); protected procedure FreeRootValue; procedure DoOnChange; virtual; public class function IsSimpleJsonValue(v: TJSONValue): boolean; inline; class function UnQuote(s: string): string; inline; class function StripNonJson(s: string): string; inline; constructor Create(AOwner: TComponent); override; destructor Destroy; override; function ProcessJsonText: boolean; function IsActive: boolean; function EstimatedByteSize: integer; property RootValue: TJSONValue read FRootValue write SetRootValue; published property JsonText: string read FJsonText write SetJsonText; property OnChange: TNotifyEvent read FOnChange write FOnChange; end; The full source code of this component and all other source code described in this paper can be downloaded from [1]. See “References” section at the end of this article. The “TJSONDocument” contains published “JsonText: string” property that can be used to assign JSON text for parsing and “RootValue: TJSONValue” public property that can be used to assign “TJSONValue” reference and generate JSON text. Assigning to any of these properties cause the other property to be updated and the “OnChange” event is fired every time the JSON stored inside the component is changed. In this way it is possible for other components of an application to be notified and refreshed. In this sense “TJSONDocument” component can be used as a JSON parser and generator as described in the original JSON RFC [2]. The public “IsActive: boolean” property returns true if TJSONDocument component contains valid JSON, or false it is empty. function TJSONDocument.IsActive: boolean; begin Result := RootValue <> nil; end;
The “TJSONObject.ParseJSONValue: TJSONValue” method is sensitive to the contents of the JSON text passed for parsing. If the string provided does not contain valid JSON text or it contains JSON text with additional whitespace characters, than it is always returning “nil” TJSONValue reference. The class function “StripNonJson” is used to remove from JSON text any non JSON characters and is implemented as follows using “TCharacter” class from the VCL “Character” unit. class function TJSONDocument.StripNonJson(s: string): string; var ch: char; inString: boolean; begin Result := ''; inString := false; for ch in s do begin if ch = '"' then inString := not inString; if TCharacter.IsWhiteSpace(ch) and not inString then continue; Result := Result + ch; end; end; The process of JSON parsing is implemented in “ProcessJsonText” method that is called as a sideeffect of assigning to “JsonText: string” published property. procedure TJSONDocument.SetJsonText(const Value: string); begin if FJsonText <> Value then begin FreeRootValue; FJsonText := Value; if FJsonText <> '' then ProcessJsonText end; end; function TJSONDocument.ProcessJsonText: boolean; var s: string; begin FreeRootValue; s := StripNonJson(JsonText); FRootValue := TJSONObject.ParseJSONValue(BytesOf(s),0);
Result := IsActive; DoOnChange; end; The “TJSONDocument” was designed to be as minimal as possible. For convenience it also surfaces “EstimatedByteSize: integer” method provided by the underlying DBXJSON implementation. This is how the “TJSONDocument” component looks at design-time inside the Delphi 2010 Object Inspector. Hide
image
TJSONTreeView Component I have always wanted to implement a JSON viewer in DelphiJ The “TJSONDocument” component is non-visual, so I needed a separate visual component that would provide graphical, tree-representation of JSON. This component should have a “JSONDocument” published property to connect both components at design-time. How JSON should be visualized? Is simple Delphi “TTreeView” component good enough or maybe I should do some fancy painting in code? Maybe I should use TVirtualTreeView component to have a tree with multiple columns? These are all good questions, so I had to do a little googling around for inspiration. There are simpler and more complex JSON viewers available in Internet. Some of them are standalone applications,
like
the
one
coded
in
.NET
and
available
at
http://jsonviewer.codeplex.com/. Other viewers are embedded at web pages likehttp://www.jsonviewer.com/ or http://jsonviewer.stack.hu/. The one that I liked the most was implemented in Java and is available as a part of the Apache Pivot project for Rich Internet Applications (http://pivot.apache.org/demos/jsonviewer.html). The one cosmetic thing that I do not like about it, is that is sorts JSON properties alphabetically and does not preserve the original ordering of object pairs. Below is the screenshot from the apache pivot web page and this is my desired TreeView-based JSON viewer functionality: Hide
image
Here we goJ I have decided to create my JSON tree view component as a descendant of Delphi VCL “TTreeView” component. A good Delphi programming practice would be to derive from “TCustomTreeView” instead to be able to decide which inherited “protected” members of a class should be declared as “published”. In my case I want the end user to have access to whole TTreeView component functionality at design-time, so I do not need to hide any inherited properties. unit jsontreeview; // … type TJSONTreeView = class(TTreeView) // … public procedure LoadJson; published property JSONDocument: TJSONDocument // … property VisibleChildrenCounts: Boolean // …
property VisibleByteSizes: Boolean // … end; Additionally to the original viewer feature set I have also added the possibility to display children counts next to every non-empty JSON object or array, and estimated byte size of a given JSON tree node. The main functionality of this component is implemented in its public “LoadJson” procedure that populates the tree view based on the content of connected “TJSONDocument” component. procedure TJSONTreeView.LoadJson; var v: TJSONValue; currNode: TTreeNode; i, aCount: integer; s: string; begin ClearAll; if (JSONDocument <> nil) and JSONDocument.IsActive then begin v := JSONDocument.RootValue; Items.Clear; if TJSONDocument.IsSimpleJsonValue(v) then Items.AddChild(nil, TJSONDocument.UnQuote(v.Value)) else if v is TJSONObject then begin aCount := TJSONObject(v).Size; s := '{}'; if VisibleChildrenCounts then s := s + ' (' + IntToStr(aCount) + ')'; if VisibleByteSizes then s := s + ' (size: ' + IntToStr(v.EstimatedByteSize) + ' bytes)'; currNode := Items.AddChild(nil, s); for i := 0 to aCount - 1 do ProcessPair(currNode, TJSONObject(v), i) end else if v is TJSONArray then begin aCount := TJSONArray(v).Size; s := '[]'; if VisibleChildrenCounts then s := s + ' (' + IntToStr(aCount) + ')'; if VisibleByteSizes then
s := s + ' (size: ' + IntToStr(v.EstimatedByteSize) + ' bytes)'; currNode := Items.AddChild(nil, s); for i := 0 to aCount - 1 do ProcessElement(currNode, TJSONArray(v), i) end else raise EUnknownJsonValueDescendant.Create; FullExpand; end; end; procedure TJSONTreeView.ProcessPair(currNode: TTreeNode; obj: TJSONObject; aIndex: integer); var p: TJSONPair; s: string; n: TTreeNode; i, aCount: integer; begin p := obj.Get(aIndex); s := TJSONDocument.UnQuote(p.JsonString.ToString) + ' : '; if TJSONDocument.IsSimpleJsonValue(p.JsonValue) then begin Items.AddChild(currNode, s + p.JsonValue.ToString); exit; end; if p.JsonValue is TJSONObject then begin aCount := TJSONObject(p.JsonValue).Size; s := s + ' {}'; if VisibleChildrenCounts then s := s + ' (' + IntToStr(aCount) + ')'; if VisibleByteSizes then s := s + ' (size: ' + IntToStr(p.EstimatedByteSize) + ' bytes)'; n := Items.AddChild(currNode, s); for i := 0 to aCount - 1 do ProcessPair(n, TJSONObject(p.JsonValue), i); end else if p.JsonValue is TJSONArray then begin
aCount := TJSONArray(p.JsonValue).Size; s := s + ' []'; if VisibleChildrenCounts then s := s + ' (' + IntToStr(aCount) + ')'; if VisibleByteSizes then s := s + ' (size: ' + IntToStr(p.EstimatedByteSize) + ' bytes)'; n := Items.AddChild(currNode, s); for i := 0 to aCount - 1 do ProcessElement(n, TJSONArray(p.JsonValue), i); end else raise EUnknownJsonValueDescendant.Create; end; procedure TJSONTreeView.ProcessElement(currNode: TTreeNode; arr: TJSONArray; aIndex: integer); var v: TJSONValue; s: string; n: TTreeNode; i, aCount: integer; begin v := arr.Get(aIndex); s := '[' + IntToStr(aIndex) + '] '; if TJSONDocument.IsSimpleJsonValue(v) then begin Items.AddChild(currNode, s + v.ToString); exit; end; if v is TJSONObject then begin aCount := TJSONObject(v).Size; s := s + ' {}'; if VisibleChildrenCounts then s := s + ' (' + IntToStr(aCount) + ')'; if VisibleByteSizes then s := s + ' (size: ' + IntToStr(v.EstimatedByteSize) + ' bytes)'; n := Items.AddChild(currNode, s); for i := 0 to aCount - 1 do ProcessPair(n, TJSONObject(v), i); end else if v is TJSONArray then begin
aCount := TJSONArray(v).Size; s := s + ' []'; n := Items.AddChild(currNode, s); if VisibleChildrenCounts then s := s + ' (' + IntToStr(aCount) + ')'; if VisibleByteSizes then s := s + ' (size: ' + IntToStr(v.EstimatedByteSize) + ' bytes)'; for i := 0 to aCount - 1 do ProcessElement(n, TJSONArray(v), i); end else raise EUnknownJsonValueDescendant.Create; end;
JSON Standalone Viewer In the next step I have used “TJSONDocument” and “TJSONTreeView” components to implement a simple JSON Viewer application. The functionality is minimal. You can clear the current contents of the JSON viewer using “Clear” button, or you can copy to clipboard a JSON text and paste it into the viewer window using “Paste” button. There is also a popup menu to control if children counts and node byte sizes are displayed or not. The application icon was created using IcoFX (http://icofx.ro/) directly from the JSON logo downloaded from the JSON home page. Below is the screenshot from Delphi JSON Viewer at runtime. Just copy some JSON text to clipboard and paste… Hide
image
TJSONParser Component In a sense the “TJSONDocument” component can be considered the implementation of Document Object Model for JSON. But what about SAX for JSON? SAX – or Simple API for XML – presents a completely different approach to document parsing. Instead of building an in-memory representation of the document, it just goes through it and fires events for every syntactical element encountered. It is up to application to process the events it is interested in. For example to find something inside a large document. Based on the “TJSONDocument” I have implemented an experimental “TJSONParser” component that implements SAX processing model for JSON. In reality the “SAX parser” for
JSON should be implemented from scratch and directly parse JSON text and fire relevant events. In my case it sits on top of the in-memory representation of JSON. The “jsonparser” unit contains the following enumerated type that lists different token types that can be found in a JSON text: type TJSONTokenKind = ( jsNumber, jsString, jsTrue, jsFalse, jsNull, jsObjectStart, jsObjectEnd, jsArrayStart, jsArrayEnd, jsPairStart, jsPairEnd ); There is also a declaration of “TJSONTokenEvent” that is fired when a JSON token is encountered: type TJSONTokenEvent = procedure( ATokenKind: TJSONTokenKind; AContent: string) of object; The “TJSONParser” class is derived from “TJSONDocument” and declared as follows: type TJSONParser = class(TJSONDocument) private FOnToken: TJSONTokenEvent; FTokenList: TJSONTokenList; procedure DoOnAddToTokenListEvent( ATokenKind: TJSONTokenKind; AContent: string); procedure DoOnFireTokenEvent( ATokenKind: TJSONTokenKind; AContent: string); public constructor Create(AOwner: TComponent); override; destructor Destroy; override; procedure FireTokenEvents; procedure BuildTokenList; procedure DoProcess(val: TJSONValue; aTokenProc: TJSONTokenProc); property TokenList: TJSONTokenList read FTokenList; published property OnToken: TJSONTokenEvent read FOnToken write FOnToken; end; The “TJSONParser” component can do two things. If you call “FireTokenEvents” public method,
it will traverse the underlying JSON document and fire “OnToken” events for every token it encounters. It is also possible to build a token list in memory that can be accessed via “TokenList” property. This could be useful if we would like to implement JSON viewer using the Virtual Tree View component that requires fast access to the underlying data structure. What was interesting during the implementation of these two methods was the fact the underlying documental traversal algorithm was the same for both firing the events and building the list. In order to avoid code duplication, I have decided to parameterize the traversal algorithm using anonymous methods. The following anonymous method signature was defined in “jsonparser” unit: type TJSONTokenProc = reference to procedure( ATokenKind: TJSONTokenKind; AContent: string); The signature of this method matches “DoOnAddToTokenListEvent” and “DoOnFireTokenEvent” private methods in the declaration of the “TJSONParser” class. The actual document traversal algorithm has been implemented inside “DoProcess” method that is called from both “FireTokenEvents” and “BuildTokenList” method in the following way: procedure TJSONParser.BuildTokenList; begin if RootValue <> nil then DoProcess(RootValue, DoOnAddToTokenListEvent); end; procedure TJSONParser.FireTokenEvents; begin if RootValue <> nil then DoProcess(RootValue, DoOnFireTokenEvent); end; procedure TJSONParser.DoOnFireTokenEvent(ATokenKind: TJSONTokenKind; AContent: string); begin if Assigned(FOnToken) then FOnToken(ATokenKind, AContent); end; procedure TJSONParser.DoOnAddToTokenListEvent(ATokenKind: TJSONTokenKind; AContent: string); begin FTokenList.Add(ATokenKind, AContent);
end; In this way we have both functionalities implemented without code duplication inside the recursive “DoProcess” method: procedure TJSONParser.DoProcess(val: TJSONValue; aTokenProc: TJSONTokenProc); var i: integer; begin if val is TJSONNumber then aTokenProc(jsNumber, TJSONNumber(val).Value) else if val is TJSONString then aTokenProc(jsString, TJSONString(val).Value) else if val is TJSONTrue then aTokenProc(jsTrue, 'true') else if val is TJSONFalse then aTokenProc(jsFalse, 'false') else if val is TJSONNull then aTokenProc(jsNull, 'null') else if val is TJSONArray then begin aTokenProc(jsArrayStart, ''); with val as TJSONArray do for i := 0 to Size - 1 do DoProcess(Get(i), aTokenProc); aTokenProc(jsArrayEnd, ''); end else if val is TJSONObject then begin aTokenProc(jsObjectStart, ''); with val as TJSONObject do for i := 0 to Size - 1 do begin aTokenProc(jsPairStart, Get(i).JsonString.ToString); DoProcess(Get(i).JsonValue, aTokenProc); aTokenProc(jsPairEnd, ''); end;
aTokenProc(jsObjectEnd, ''); end else raise EUnknownJsonValueDescendant.Create; end; The “TJSONParser” component can be used as a starting for arbitrary JSON processing at the lowest level of actual JSON text tokens. Delphi anonymous methods are really coolJ
Summary JSON is currently probably the most important data interchange format in use. Its simplicity makes it easy to process and information encoded with JSON is typically smaller than using XML. Over the years XML has become the whole family of specifications and it is not a trivial task to implement fully compliant XML parser from scratch. Delphi 6 was the first commercial IDE on the market to introduce support for XML SOAP web services. Delphi 6 also introduced TXMLDocument component and XML Data Binding Wizard to make it easier to work with XML. In the world of JSON slowly emerges the equivalent of XML Schema for JSON, which tries to create a meta representation of JSON. On the JSON home page you can find a reference to a draft version of IETF RFC “A JSON Media Type for Describing the Structure and Meaning of JSON Documents” [8]. This is still pending feedback but in future could be a starting point for implementing a “Data Binding Wizard for JSON”. In this article I have described a JSON Viewer application implemented with Embarcadero Delphi XE. The source code that accompanies this paper is organized in the form of two packages for Delphi components – one runtime and one design-time – and the “djsonview”: Delphi VCL Forms application that implements the Delphi JSON Viewer. The full source code described in this paper can be downloaded from [1].
