Showing posts with label NoSQL. Show all posts
Showing posts with label NoSQL. Show all posts

Wednesday, March 19, 2014

Link to Elastic with ElasticLINQ

Brad Wilson - Getting Started with ElasticLINQ

Jim Newkirk and I have been doing for 7 years now (and for Jim, NUnit for many years before that). You could say that open source is part of our blood, and when we left Microsoft, we made sure that open source would continue to be part of our daily efforts at Tier 3.

Fast forward 15 months: Tier 3 has been acquired (and is now the CenturyLink Cloud Development Center), and our first major open source effort Iron Foundry has been accepted into the Cloud Foundry Incubator project. Lots of great developers are working to ensure that you can write .NET code against a Platform-as-a-Service stack that doesn't lock you into a specific vendor.

Today we are proud to announce our second major open source effort: ElasticLINQ.

What is ElasticLINQ?

One of the major challenges when writing distributed software is how to distribute the data. When I started here 15 months ago, we had 4 data centers, and plans to expand into several more over the coming year. The data was being stored primarily in Microsoft SQL Server. As our data center footprint grew, it was becoming clear that centralized data storage was not going to scale with us. Having islands of data means that your application (and your users) can end up spending a lot of time waiting for data requests to go halfway around the world; and if there are any network glitches along the way, you might even fail to get the data entirely.

Almost right away we started evaluating alternatives that would let us keep all the data locally. We decided to use Couchbase as our primary data store, based on its extremely strong Cross Data-Center Replication (XDCR) capabilities. Many object data storage systems end up paired with an index engine for comprehensive searching capabilities. Couchbase provides an indexing integration solution with Elasticsearch, a horizontally scalable wrapper around Lucene.

The Lucene query syntax is based on JSON; ElasticSearch documents are also stored as JSON. Our developers, steeped in the worlds of .NET and SQL Server, were much more comfortable using the Language Integrated Query (LINQ) functions introduced in .NET 3.5.

ElasticLINQ bridges these two worlds by letting us query Elasticsearch using LINQ, and have the results projected into CLR types. We enlisted the expertise of Damien Guard (of Attack Pattern), who worked on both the LINQ to SQL and Entity Framework teams, to do the initial version of ElasticLINQ for us.

How do I use ElasticLINQ?

Connection and Context ...

Querying with the Context ...

Full-text searching ...

Custom queries with ElasticMethods ...


Custom queries and projections with ElasticFields ...

What's next?

This is v1.0 software, so we have a lot left that we can do. We've just recently started using this in our production code, and we are constantly finding new things we want to support. We expect you will come up with things we never dreamed of, too.

We are excited for the community to start using and contributing to ElasticLINQ. The Github site is a work in progress. Soon we will get documentation posted to the Wiki pages on the site, and get a real home page set up. We are anxiously awaiting the first community contributed bugs, Wiki edits, and pull requests.

We hope you love using ElasticLINQ as much as we do!


Got to love LINQ Love! :)

Now if only I knew something, anything about Elastic... :/

Tuesday, October 22, 2013

patterns & practices: Data Access Guidance code drop for the "Data Access for Highly-Scalable Solutions: Using SQL, NoSQL, and Polyglot Persistence" eBook now available

patterns & practices: Data Access Guidance - Data Access Guidance 2013

Recommended Download

Patterns and Practices Data Access Guide.Source.Exe
application, 1412K, uploaded Today - 5 downloads

Other Available Downloads

application, 895K, uploaded Today - 8 downloads

application, 427K, uploaded Today - 7 downloads

documentation, 8903K, uploaded Today - 8 downloads

application, 4020K, uploaded Today - 3 downloads

Release Notes

This is the 2013 release of Data Access Guidance.

The documentation for this RI is also available on MSDN:
Data Access for Highly-Scalable Solutions: Using SQL, NoSQL, and Polyglot Persistence:

patterns & practices: Data Access Guidance

Data Access Guidance project covers data access for SQL and NoSQL databases as well as some architectural and design decisions involved in using them.

This project is guidance around data access using relational databases, as well as NoSQL databases. It describes the various types of data persistence technologies, as well as when, how, and why they should be used. The guidance will show how various persistence types can be used together to create a highly-scalable system, as well as illustrating the challenges that arise when moving to a polyglot persistent system.
Getting started

Here's a snip from the readme.pdf;

Example Summary
This document describes how to configure the Visual Studio example solution that accompanies the guide Building Highly-Scalable Solutions for Storing and Retrieving Data Using SQL and NoSQL Databases. This guide focuses on the common issues that you may encounter when building applications that need to store data in a variety of different data stores, such as a relational database, a key-value store, a document database, or a graph database.

This document contains the following sections:
 About the Example Solution describes the high level structure of the sample application.
 Before You Start describes the procedures that you must follow before configuring and running the application.
 Supported Scenarios describes how to build and run the application in each of the scenarios described in the guide.
 Configure the Application describes how to modify the configuration to support each of the scenarios described in the guide.
 Check the Sample Functionality describes how to verify that the application is running correctly.

About the Example Solution
The example solution implements a shopping application that enables customers to browse products and place orders. The application runs by using the web. Orders are handled by a separate order service, and for completeness the example solution includes a simulation of this order service.

The Shopping application is a polyglot system that stores the data it uses in a variety of different repositories. Each repository was selected to meet the business requirements of a specific scenario in the application:
 Customer information is stored in a relational database.
 Shopping cart information is stored in a key/value store.
 Product information is stored in a document database.
 Active order information is stored in a relational database.
 Order history information is stored in a document database.
 Product recommendations are stored in a graph database.



Not a simple project, which given all the data sources, kinds of makes sense... I mean a 22 page readme just to get the code working...


Related Past Post XRef:
No need to say no to NoSql - "Data Access for Highly Scalable Solutions: Using SQL, NoSQL, and Polyglot Persistence" Free eBook and code....

Wednesday, October 02, 2013

No need to say no to NoSql - "Data Access for Highly Scalable Solutions: Using SQL, NoSQL, and Polyglot Persistence" Free eBook and code....

Andrew Oakley's Blog - Just Released – Data Access for Highly Scalable Solutions: Using SQL, NoSQL, and Polyglot Persistence

All applications use data, and most applications also need to store this data somewhere. In the world of business solutions, this often meant creating a relational database. However, relational technology is not always the best solution to meet the increasingly complex data-processing requirements of modern business systems, especially when this processing involves storing and retrieving massive amounts of data.

The advent of NoSQL databases has changed the way in which organizations have started to think about the way in which they structure their data. There is no standard definition of what a NoSQL database is other than they are all non-relational. They are less generalized than relational databases, but the driving force behind most NoSQL databases is focused efficiency and high scalability.

The downside of NoSQL is that no single database is likely to be able to support the complete range of business requirements mandated by your applications. How do you select the most appropriate database to use, or should you remain with the relational model? A modern business application is not restricted to using a single data store, and an increasing number of solutions are now based on a polyglot persistent architecture. The key to designing a successful application is to understand which databases best meet the needs of the various parts of the system, and how to combine these databases into a single, seamless solution.

This guide helps you understand these challenges and enables you to apply the principles of NoSQL databases and polyglot persistent solutions in your own environment.

To help illustrate how to build a polyglot persistent solution, this guide presents a case study of a fictitious company faced with building a highly scalable web application capable of supporting many thousands of concurrent users.


How to get it?

The source code for the associated reference implementation can be found on the Microsoft Download Center.

You can view the documentation on MSDN or you can download a PDF from the Microsoft Download Center.  The written guidance is also available for purchase on Amazon as a printed book or a Kindle download.

Book Download: Data Access for Highly-Scalable Solutions: Using SQL, NoSQL, and Polyglot Persistence

This guide describes how to design and build applications and services that can take best advantage of SQL and NoSQL databases by combining them into a polyglot solution. It provides an end to end walkthrough of a business application that uses SQL Server in conjunction with a variety of NoSQL databases, showing how the designers selected the databases to closely match the business requirements.

Version: 1

Date Published: 9/27/2013

DataAccessForHighlyScalableSolutions.epub, 3.9 MB

DataAccessForHighlyScalableSolutions.pdf, 8.7 MB

This is a PDF version of the on-line guidance available on MSDN Data Access for Highly-Scalable Solutions: Using SQL, NoSQL, and Polyglot Persistence

This guide describes how to design and build applications and services that can take best advantage of SQL and NoSQL databases by combining them into a polyglot solution. It provides an end to end walkthrough of a business application that uses SQL Server in conjunction with a variety of NoSQL databases, showing how the designers selected the databases to closely match the various business requirements. This guide has an accompanying reference implementation of an online ordering system. The sample code illustrates how to implement a polyglot solution that stores data in a variety of SQL and NoSQL databases, and how to decouple the data access logic from the business logic of the application.


    I like the idea of using the right tool for the right problem. I think this kind of model is one that we'll see much more of... (So I better start wrapping my head around it now... ;)

    Friday, October 26, 2012

    Sacha Barber shares his WPF/RavenDB/MVVM Diagram Designer application

    CodeProject - MVVM Diagram Designer



    A while back a user called "sucram (real name Marcus)"  posted a series of articles here about how to create a diagram designer using WPF. Sucrams original links are as follows:

    I remember being truly blown away by this series of articles, as they showed you how to do the following things:

    • Toolbox
    • Drag and Drop
    • Rubber band selection using Adorners
    • Resizing items using Adorners
    • Rotating items using Adorners
    • Connecting items
    • Scrollable designer surface, complete with zoombox

    WOW that sounds fantastic, sounds exactly like the sort of things you would need to create a fully functional diagram designer. Well Yeah, its was and still is, but........the thing is I have used WPF a lot, and trying to use the code attached to sucrams series of article in WPF just wasn't that great. He had taken a very control centric view, in that everything was geared around adding new controls and supplying static styles for said controls.

    In reality it was more like working with a Win Forms application. Not that there is anything wrong with that, and I really truly do not mean to sound ungrateful, as that could not be further from the truth, without that original series of articles it would have taken me a lot longer to come up with a working diagram designer that I was happy with. So for that I am truly grateful, thanks sucram you rock.

    Anyway as I say sucrams original codebase took a very control centric point of view, and added controls using code behind, and held collections of items directly in the diagram surface control. As I say if that is what you want cool, however, it was not what I wanted. What I wanted was

    • All of the features of curams original code (actually I didn't want any rotating of items, or resizing of items)
    • A more MVVM driven approach, you know allow data binding of items, delete of items via ICommand etc. etc.
    • Allow me to control the creation of an entire diagram from within a single ViewModel
    • Allow for complex objects to be added to the diagram i.e. ViewModels that I could style using DataTemplate(s). Sucrams original code only allowed simply strings to be used as a DataContext which would control what ImageSource an Image would use to show for a diagram item. I needed my items to be quite rich and allow popups to be shown and associated with the diagram item, such that the data related to the diagram item could be manipulated
    • Allow me to save the diagram to some backing store
    • Allow me to load a previously saved diagram from some backing store

    To this end I have pretty much completely re-written sucrams original code, I think there is probably about 2 classes that stayed the same, there is now more code, a lot more, however from an end user experience, I think it is now dead easy to control the creation of diagrams from a centralized ViewModel, which allows a diagram to be created via well known WPF paradigms like Binding/DataTemplating.

    For example this is how the attached DemoApp code creates a simple diagram that is shown when you first run the DemoApp:


    What Does It Look Like

    This is quite interesting, as if you look at the screen shot below and compare that to the final article that sucram produces you probably won't see any difference, which as I previously stated was intentional. I think sucram really nailed it, I just wanted a more WPF style codebase, one that supported Binding etc. etc., so yeah I must admit you could easily look at this screen shot and think "Bah humbug......this is exactly the same", well yes visually speaking I guess it is, however the code is very very different, and the way in which you work with the diagram is very different. Anyway enough chat here is a screen shot.



    This project is a demonstration project, and is a good example of how to create your own diagram designer. It is a fully functioning demo, and also demonstrates persisting/hydrating using RavenDB which is a NoSQL document database (as I could not be bothered writing loads of SQL)

    Persistence common classes, used by DemoApp

    I decided to use RavenDB for persistence which is a NoSQL database, that allows raw C# objects to be stored. I decided to do this, as I really couldn't be bothered to create ALL the SQL to save/hydrate diagrams, and I just wanted to get something up an running ASAP
    Though if you use SQL server/MySQL etc. etc., it should be pretty easy to create the stored procedures/data access layer that talks to your preferred SQL database.

    This project contains the core classes that are needed to create a diagram in WPF

    How Do I Use It In My Own Applications

    This section will talk you through how to create a diagram in your own application. It assumes the following

    • That you want to use WPF things like Binding/DataTemplating/MVVM
    • You actually want to persist / hydrate diagrams to some backing store (Like I say I chose to use RavenDB which is a no sql document database, but if this is not for you, it should be pretty easy for you to craft your own data access layer talking to your preferred SQL backend)

    If you want to create your own MVVM style diagram designer, I have broken it down into 7 easy steps, as long as you follow these 7 steps to the letter you should be just fine. There is also a working example of these 7 steps by way of the attached DemoApp project code, so you can examine that whilst reading this text, so hopefully you will be ok.

    Use It Step 1 : Creating The Raw XAML


    Have you clicked through yet? Have you seen the depth and how long this article is? Yeah, wow.

    I dig that he used RavenDB. I've been following that product but never used it (or think I've seen in used in a sample/demo I've downloaded). Then of course there's the MVVM goodness and everything else he's showing off. This should keep you code-geeking for a while...:)

    Tuesday, May 22, 2012

    Know NoSql? Here's a .Net dev focused look at Redis, Mongo and Raven

    CodeProject - Document Databases : A look at them


    I don't know how many of you work with SQL. Loads? Well I do, as we know it's a relational database which we can store um well relational SQL data types in, such as INT/CHAR/NVARCHAR etc etc, I am sure you all know what I am talking about and have used relational databases loads in the past.

    Do you think there are other sort of databases out there? No? Well actually there are some different types of databases other than relational databases, such as

    • Flat file
    • Object
    • NoSQL / Document / KeyValue pair

    Now I do not profess to know much about Flat file or Object databases per se, but I have spent some time evaluating and getting to know some of the newer Document databases. In this article I have chosen to look at 3 different Document databases, which I have created demos for in the code attached to this article, but before we go on to talk about each of them and how to get started with them, let's just spend a bit of time talking about the gaining popularity of these document databases.

    So I have stated that this article will talk about document databases, but what are these document databases, and why might you want to use one.


    Now there are loads and loads of document databases out there, far too many for me to go through, for my initial evaluations I chose to look at a few based on what I perceived to be the best attributes, such as

    • Features
    • Ease of use
    • Reputation
    • .NET driver availability (I am a .NET developer these day, so this article is about using .NET with the relevant document database)

    With those list of attributes, I ended up with quite a large list, which I whittled down further to end up with 3 document databases, which I will talk about in this article.

    1. Redis
    2. Raven
    3. Mongo

    Do not expect to reach the end of this article and be an expert in document databases, but I would hope by the end of reading this article you will be able to understand how they work (kinda) and would be able to carry on using them and finding any remaining answers you need by yourselves.



    The .Net dev focus of this article is what caught my eye (funny that). I've not played in the NoSql space yet, but I do have a project in mind that might be a good fit...

    Tuesday, April 24, 2012

    DeNSo DB - A new C# based NoSQL with InProc, Service or Mesh modes

    DZone - Introducing DeNSo DB - A New NoSQL Database written in C# for a .Net Environment

    "DeNSo DB is provided in its entirety on Github, where you can get a description of its usage, features and theory. The github site has yet to be really fleshed out, and I'm sure readers would be particularly interested in seeing some answers to the question at the bottom: Why should I use it? Maybe you can start formulating your own opinion on that matter by taking a look at this brief summary of what exactly DeNSo DB is.

    Right now, DeNSo DB is contending with a few well-established NoSQL databases like MongoDB, RavenDB, and HBase, each of which cater to specific usage. HBase, for example, is a Hadoop database, so it works well with processing Big Data. DeNSoDB currently has a few usages described on the github page:


    DeNSo DB

    DensoDB is a new NoSQL document database. Written for .Net environment in c# language.
    It's simple, fast and reliable.

    You can use it in three different ways:

    1. InProcess: No need of service installation and communication protocol. The fastest way to use it. You have direct access to the DataBase memory and you can manipulate objects and data in a very fast way.

    2. As a Service: Installed as Windows Service, it can be used as a network document store.You can use rest service or wcf service to access it. It's not different from the previuos way to use it but you have a networking protocol and so it's not fast as the previous one.

    3. On a Mesh: mixing the previous two usage mode with a P2P mesh network, it can be easily syncronizable with other mesh nodes. It gives you the power of a distributed scalable fast database, in a server or server-less environment.

    You can use it as a database for a stand alone application or in a mesh to share data in a social application. The P2P protocol for your application and syncronization rules will be transparet for you, and you'll be able to develop all yor application as it's stand-alone and connected only to a local DB.


    1. Journaled
    2. Built using Command Query Responsibility Segregation pattern in mind.
    3. Store data as Bson-like Documents.
    4. Accessible via Rest
    5. Accessible via WCF
    6. Peer to peer syncronization and event propagation enabled.
    7. Pluggable via Server Plugin.
    8. Use Linq syntax for queries.

    The Theory

    A document database is a Database where data are stored as they are. Usually there is no need to normalize or change their structure, so it's optimal for domain driven design, because you'll not need to think about how to persist your domain.

    A document is stored in BSon-like format A document is self contained, and all you needed to understand it's information are in the document itself. So you will not need additional query to understand foreign key or lookup table datas.

    You can have collections of document. Collection are not tables, you can think to collections as sets of documents, categorized in the same way. A collection is schema-free and you can store different kind of documents in the same collection.

    Every document in every collection is classified using a Unique Identifier. The DB will use the UID to store and to access every document directly.

    You can store your data in DeNSo using a Bson document or, more easly you can store your entities in DeNSo.

    DeNSo has a Bson De/Serializer able to traduce virtually every entity in a BSon Document and vice-versa." [GD: leached in full]

    I've yet to jump on the NoSQL boat, but have been thinking about it for a couple projects and like that this one is OSS and written in C# (not that I'm saying the others aren't, just that the C# caught my eye...)

    Thursday, September 01, 2011

    "Intro to NoSQL" or "Noodling NoSQL" or "No NoSQL? Now know NoSQL...")

    FRD - Intro to NoSQL


    A Brief Overview:

    Databases have been a major part of software since the earliest days of computing. Hence, it didn’t take long for Database Management Systems (DBMSes) to be developed and quickly become a staple in software development. Their history can be traced back to the 1960′s, but most of the DBMSes still in use today trace their origins to Relational DBMSes (RDBMSes) originally developed in the 1970′s.
    However, there is a new group of DBMSes that have gained support and relevance in the last few years, all sharing certain characteristics and collectively known as NoSQL DBMSes.

    The term “NoSQL” was originally used in 1998 to describe a new lightweight open-source relational DBMS that did not expose an SQL interface. The name was then reprised and used 10 years later by Eric Evans (of Rackspace) and Johan Oskarsson (of for the NoSQL meetup/conference focusing on the topic of “open-source, distributed, non relational databases“.
    Clearly the name NoSQL had grown past its initial meaning (note how the original NoSQL DBMS is indeed a relational system).



    Any blog post that begins with a picture like that deserves a second look... :P


    You'd have to be living under a rock (not that there's anything wrong with that) to not have at least heard the term, "NoSQL." But you might be in the boat (cough... with me... cough) wondering what it "is.". I mean I know, or can pretend to, what it means and implies, but my knowledge lacks depth (cough... to say the least... cough) (cough... I really have to have this cough looked at... cough)

    I dug the way this article presented NoSQL, that it did so where I learned some intelligent sounding words, while not hurting my brain too much and all the while learning about NoSQL too.

    (via Jason Haley - Interesting Finds: September 1, 2011 )