This page was converted from my old blog and hasn’t been reviewed. If you see an error please let me know in the comments.
SpringXD is a new project which simplifies the development of Big Data Applications. SpringXD’s ability to stream data between two different modules. In this example, we will use an HTTP module as a data source and a GemFire module as a data sink.
One Component of Big Data Applications is Pivotal’s GemFire. SpringXD has some built-in connections to GemFire which allow you to get up and running quickly.
Before reading this guide I would recommend that you have a working SpringXD install. The best place to get going with SpringXD is the getting started guide over at spring.io
The data we will be posting is a Customer. The JSON that represents this customer looks like this.
{"firstname":"BOB","lastname":"JONES"}
{"firstname":"JOE","lastname":"SMITH"}
{"firstname":"MARY","lastname":"JANE"}
Start the Gemfire Server
SpringXD comes with a scaled down GemFire server if you are on a Mac and installed SpringXD via BREW the command to start Gemfire server will already be on your path. The configuration file I supply has a CacheWriter that will log data placed into the server so that we can verify that our SpringXD stream is working correctly.
The GemFire instance will be booted using a simple context file.
- Create a temporary working directory for Gemfire in a location of your choosing.
- Copy the above xml file to your working directory
- Startup GemFire
gemfire-server cq-demo.xml
Start the SpringXD Single Node instance
Once your GemFire server starts up open another terminal window and startup your SpringXD server if it isn’t started already
xd-singlenode
Start the SpringXD Shell
Once your SpringXD instance starts up open another terminal window and start the SpringXD Shell.
xd-shell
Starting a Stream in SpringXD
Once connected to the SpringXD Shell, the next step is to start a stream to accept HTTP data and store it in GemFire.
SpringXD includes two modules that allow this to be done without writing any code.
- The HTTP Module accepts data posted from an external source.
- The GemFire-Json-Module accepts a JSON object, converts it to PDX and saves it in GemFire.
The SpringXD shell is used to connect these two modules into a Stream.
stream create --name stocks --definition "http --port=9090 | gemfire-json-server --regionName=Customers --keyExpression=payload.getField('lastname')" --deploy
The above stream pipes data from the http module to the GemFire server running on localhost[40404]
Post data into GemFire using the following command
http post --target http://localhost:9090 --data {"firstname":"BOB","lastname":"JONES"}
If everything is working properly you should see the posted data showing up in the terminal window for GemFire.
Last modified on 2014-09-18