Link to Elastic with ElasticLINQ

Wednesday, March 19, 2014

Link to Elastic with ElasticLINQ

Brad Wilson - Getting Started with ElasticLINQ

Jim Newkirk and I have been doing xUnit.net for 7 years now (and for Jim, NUnit for many years before that). You could say that open source is part of our blood, and when we left Microsoft, we made sure that open source would continue to be part of our daily efforts at Tier 3.

Fast forward 15 months: Tier 3 has been acquired (and is now the CenturyLink Cloud Development Center), and our first major open source effort Iron Foundry has been accepted into the Cloud Foundry Incubator project. Lots of great developers are working to ensure that you can write .NET code against a Platform-as-a-Service stack that doesn't lock you into a specific vendor.

Today we are proud to announce our second major open source effort: ElasticLINQ.

What is ElasticLINQ?

One of the major challenges when writing distributed software is how to distribute the data. When I started here 15 months ago, we had 4 data centers, and plans to expand into several more over the coming year. The data was being stored primarily in Microsoft SQL Server. As our data center footprint grew, it was becoming clear that centralized data storage was not going to scale with us. Having islands of data means that your application (and your users) can end up spending a lot of time waiting for data requests to go halfway around the world; and if there are any network glitches along the way, you might even fail to get the data entirely.

Almost right away we started evaluating alternatives that would let us keep all the data locally. We decided to use Couchbase as our primary data store, based on its extremely strong Cross Data-Center Replication (XDCR) capabilities. Many object data storage systems end up paired with an index engine for comprehensive searching capabilities. Couchbase provides an indexing integration solution with Elasticsearch, a horizontally scalable wrapper around Lucene.

The Lucene query syntax is based on JSON; ElasticSearch documents are also stored as JSON. Our developers, steeped in the worlds of .NET and SQL Server, were much more comfortable using the Language Integrated Query (LINQ) functions introduced in .NET 3.5.

ElasticLINQ bridges these two worlds by letting us query Elasticsearch using LINQ, and have the results projected into CLR types. We enlisted the expertise of Damien Guard (of Attack Pattern), who worked on both the LINQ to SQL and Entity Framework teams, to do the initial version of ElasticLINQ for us.

How do I use ElasticLINQ?

Connection and Context ...

Querying with the Context ...

Full-text searching ...

Custom queries with ElasticMethods ...

Custom queries and projections with ElasticFields ...

What's next?

This is v1.0 software, so we have a lot left that we can do. We've just recently started using this in our production code, and we are constantly finding new things we want to support. We expect you will come up with things we never dreamed of, too.

We are excited for the community to start using and contributing to ElasticLINQ. The Github site is a work in progress. Soon we will get documentation posted to the Wiki pages on the site, and get a real home page set up. We are anxiously awaiting the first community contributed bugs, Wiki edits, and pull requests.

We hope you love using ElasticLINQ as much as we do!

...