Southern BITS of Technical Goodness

DDD has changed our modeling approaches. Eric Evans book has lead to a variety of innovative ideas such as DCI (Data Context Integration) and recently Erick Flemming introduced me to CQRS. I’ve spent the past few days running around the web consolidating a resource packet. Here are the links I’ve read and recommend:

 

Unshackle Your Domain

: Greg Young > InfoQ Link : Nice video but long (1 hour)

 

Clarified CQRS

: Udi Dahan’s righteous post on his blog here. Nice graphic.

 

CQRS à la Greg Young

An even more detailed / concrete post by Mark Nijhof that can be found here http://elegantcode.com/2009/11/11/cqrs-la-greg-young/comment-page-1/#comments. He ahs great hand drawn diagrams (previewed below, but the code is the best part… real working example)

  > He has a working example of a banking application (winforms) here: http://github.com/MarkNijhof/Fohjin

> And a video on vimeo here: http://vimeo.com/7838858

 

Domain-Driven Design (the book)

Don’t forget Eric Evans DDD book that you can find here:http://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215/ref=sr_1_1?ie=UTF8&s=books&qid=1270092554&sr=8-1

 

Other articles & frameworks

Sadly there are currently no .NET CQRS frameworks I could find. That said there are some Java. The front runner seems to be.

 

axonframework : CQRS Framework for Java

They have great documentation and a similar diagram as Mark (minus a command bus)

http://code.google.com/p/axonframework/

On InfoQ: http://www.infoq.com/articles/cqrs_with_axon_framework

 

 

EventSourcing & Snapshots

http://jonathan-oliver.blogspot.com/2009/03/event-sourcing-and-snapshots.html

Asynchronous Event Sourcing using Actors (SCALA)

Article : http://jonasboner.com/2009/02/12/event-sourcing-using-actors.html

alan.huffman .NET, Architecture, CQRS, DDD, Design & Features .NET, CQRS, DDD

In order to create user centric applications / solutions, we must keep users in mind. One of the best ways to do that is by designing software using “personas” – fictitious yet concrete representational users.

 

Pragmatic Personas: Putting the User back in User Stories

Presented by Jeff Patton on Dec 23, 2009 [ INFOQ ] [ LINK TO VIDEO ]

Summary
Jeff briefly reviews the different ways that software is currently built and then describes how to create and use user personas to design and build software that has a better user experience. Jeff walks us through how to collaboratively build a user persona, what a user persona should include, and how to use these personas to write user scenarios that end up as user stories wit.

 

 

What are personas?

Wikipedia has a nice entry: http://en.wikipedia.org/wiki/Persona_%28marketing%29

Personas are fictional characters created to represent the different user types within a targeted demographic that might use a site or product. Personas are useful in considering

1. the goals,
2. desires,
3. and limitations

of the users in order to help to guide decisions about a product, such as features, interactions, and visual design.

According to this blog [darmano] personas:

Personas often combine narratives and sometimes scenarios that often go into great detail to paint a plausible profile which looks at a person’s motivations, goals, mindset, wants, needs, desires etc.  And often times, personas are often cross-channel—taking a holistic look at the entire consumer experience.

[boxesandarrows] : “Personas and scenarios tell honest stories that are sculpted from diverse and comprehensive sets of data.”

What characteristics are included in a persona?

: http://www.usability.gov/analyze/personas.html

A persona usually includes:

• a name and picture
• demographics (age, education, ethnicity, family status)
• job title and major responsibilities
• goals and tasks in relation to your site
• environment (physical, social, technological)
• a quote that sums up what matters most to the persona with relevance for your site

 

Why personas are important:

http://www.hceye.org/HCInsight-Nielsen.htm

• Personas put a face on the customer. Some persona programs give people names so you can refer to them and see them in a physical representation. The agency Organic creates persona rooms where their people live so the project team can become fully immersed.
• Personas remove the tendency to think of yourself as the customer. You have to step back and this gives you the structure to do so.
• Act as a guide throughout the process of developing marketing communications programs, cross mediums (print, digital, outdoor, TV, etc.).
• Keeps designers, copywriters, programmers on track and avoids waste by remaining focused on the customer.

 

A 10 Step Process to Personas by Lene Nielsen, Ph.D.

http://www.hceye.org/HCInsight-Nielsen.htm

Lene Nielsen wrote her Ph.D. thesis "Engaging Personas and Narrative Scenarios" in 2004. She is part-time assistant professor at Center of Applied ICT, at Copenhagen Business School
and part-time usability consultant at Snitker & Co.

Do personas work?

Use of Personas Boosts Conversion by 400% [link]

think big by starting small

Steve Franzman, founder of Detoxologie.com, a client who used personas to boost conversion by 400%, and get a 2 to 1 return on a floundering Pay-Per-Click campaign.

 

Personas Drive Universal Orlando Site Revamp [link]

Universal Orlando’s marketing executives had spent several years tweaking the theme park’s online presence for incremental gains in conversion…. After a period of testing and conversion analysis, Universal Orlando sketched … personas, each with a different set of motivators and travel needs. … So how’s it working? …

1. new site was living up to its expectations,
2. beating the old version on 30 out of 40 measures, including likeability….
3. Online ticket purchases, a key metric for the resort, are up almost 80% year to date.

 

Personas work because they tell stories.

http://www.wqusability.com/articles/personas_storytelling.html

Stories are part of every community. They communicate culture, organize and transmit information. Most importantly, they spark the imagination as you explore new ideas. They can ignite action.

alan.huffman Architecture, Business, Design & Features Design, Personas, User Centric Design

Our business has grown quickly, from $1M in annual revenue a few years ago to almost a hundred times that now. In that time, I’ve moved from web based Java development to .NET rich client development.

The company I work for has used client server applications that use Access front end with a SQL Server back end. Access is written in VBA using forms over data in a procedural programming model.

I’ve argued the value of OOP and best practices, but my success has been gated by one thing: SPEED TO DEVELOPMENT. Why?

IT (geeks) provide what value?

I used to believe we generated profit. I used to measure my value in added revenue, but soon I realized that upper management sees IT as a cost – waste. Their goal was to minimize cost and maximize profit (or in a LEAN way of thinking, reduce waste).

IT is WASTE – we are an expense not revenue.

Don’t believe me? See your paycheck for proof. I hate this idea, but that’s the way that upper management often sees us (geeks).

So consider the two perspectives that geeks and business have. We see this:

But business owners see this:

 

As Dennis Hopper said in True Romance If that’s true, am I lying?

So what are you (GEEK) going to do about it?

We’re right. We know we’re right. We can’t be wrong. Right?!

So why in the hell are they saying big gray blobs and BIG dollar signs when we want them to see…

How to do that:

1. Reference prominent companies (& geeks within) that use best practices. Business values experience and most importantly they value SUCCESS & FAILURE. If you can show that following good practices is what successful companies do, and that unsuccessful companies have failed for not following best practices.
2. Does business want it RIGHT or right now? Software projects fail. They have failed time and time again. Getting software implementations right takes time. Moving too quickly only results in failure and rework. You can only move as fast as possible but no faster (see Einstein: Make things as simple as possible but no simpler).
3. Ask questions: Is this a long term investment or short term? If business is seeking a POC (proof of concept) or the financial/business climate is such that they need to prove something out or “just get it done” then perhaps fast is best. But if the project is a long term investment, then they should do it right. Pay upfront for good architecture, project methodologies and infrastructural support.
4. Be Reasonable. Understand business. You want business to understand you, but you need to understand business. Why should they care about your best practices when you don’t give a damn about their profitability or time to market. Remember IT is an enabler – IT is waste, help them reduce waste.
5. Best practices lead to flexibility & more responsive, cost effective solutions: Face it, best practices are going to cost more to develop. Forms over data are fast. MVP / MVC / MVVM with repository patterns and asynchronous enterprise service buses mocked out and developed using TDD just take longer. So why would business want that? If you don’t have an answer, don’t expect business to buy in.
• Design patterns increase flexibility which allow IT to respond faster to business.
• Test Driven Development decrease mistakes and bugs and leads to automated regression testing (decreased testing costs)
• Electronic Service Buses formalize application boundaries which allows business to swap out new technologies, vendors and applications as they become available thus decreasing cost and increasing IT responsiveness.

Summary:

If you can’t sell what you believe in, then you need to work on the pitch. Being a geek doesn’t omit us from having to persuade people to accept our beliefs.

IT is a sort of religion. It’s mysterious, poorly understood (even by us), and we constantly are trying to understand a thing that is far greater than ourselves – something we’ll never fully comprehend.

In the end, all a geek can do is convey what we believe in using the language of business.

alan.huffman Architecture, Business, Business Growth, Development Infrastructure, Projects Best Practices, development, leadership, management

(when migrating from SQL Designer)

History:

For the past 1.5 years our team has developed HIT (Healthcare IT) LOB (line of business) software using DLINQ SQL Designer. The experience has been pleasant from a tooling perspective except for the reasons that are frequently blogged about, namely DBDesigner results in SVN conflicts when 2 people make changes to the model.

We have a few large DB’s – one is about 9 years old and has hundreds of tables (600+) and even more views plus hundreds if not thousands of SPROC’s (many of which are complex)

We have 3 such DB’s. (each w/ their own pain)

Consider a common use:

More than 1 developer is making changes to a shared relational database (adding/changing tables, sprocs or functions).  Changes are made by Developer A & Developer B and then both try to check in. SVN detects a conflict – given that the code is generated, resolving the conflict is *not* easy to resolve.

Primary Issue:

When a checking in DLINQ Designer DLinq DBML generated files a conflict occurs.

Solution? Move from using the DLINQ Designer to using SQLMetal. (We could use Entity Framework, but that would require significant changes. So we want to stick with DLINQ (LINQ to SQL)).

SQLMetal does not generate what SQL Designer does!

Obviously, if you are on a green field project, this is not as big of an issue. If you are on an existing (brown field) project that has used SQL Designer, this hurts (a lot).

A) SQLMetal does not handle complex stored procedures (in same way)

What is a complex stored procedure?

1. Uses control flow logic (If/Else) that results in different result sets.
2. Uses temporary tables (#TmpTable )

#1 results in a IMultipleResults being returned by SQLMetal even though SQL Designer results in a ISingleResult. So migrating from SQL Designer to SQLMetal results in compilation errors.

#2 SQLMetal does not generate these sort of sprocs (at all) though SQL Designer does (ugh!)

#2 is particularly difficult to cope with, assuming you need a #Temp Table. SQLMetal just isn’t going to work.

SET FMTONLY OFF;  — explicitly in the SPROC

http://msdn.microsoft.com/en-us/library/ms173839.aspx

– what does this do? Results in the **ACTUAL** execution of the sproc and not just returning the meta data about what the SPROC result will be.

– in other words, when you run SQLMetal, your sprocs (that have FMTONLY OFF) will be executed. There could be side effects (assuming your SPROC does something & doesn’t just return information)

– This was *NOT* an option for us

B) SQLMetal does not support manual tweaking of DBML as SQL Designer does.

1. Can not add associations between DTO (Data Transfer Objects) that do *not* exist in DB. [ SQL Designer allows this ]
2. Can not map a SPROC to a DTO/Entity (easily done in DBML Designer by simply dragging the SPROC to an existing entity (table)

C) Not easy to change the inheritance hierarchy of the generated DataContext?

If you need to change the class that the datacontext inherits from the change must be made each time SQLMetal is run.

e.g.

MyDataContext  : System.Data.Linq.DataContext // generated by SQLMetal each time

// but we want

MyDataCongext : BaseDataContext

// where

BaseDataContext : System.Data.Linq.DataContext

D) SQLMetal DBML or Code generation is VERY SLOW for large DB (15 min !)

Can you imagine adding a table and then waiting 15 minutes before you can start coding using that new table b/c SQLMetal takes that long to generate the DBML or DataContext.cs?

Me neither b/c this is totally unacceptable

E) SQLMetal generates the entire database model! (ugh)

The only filtering that can be done using SQLMetal is at the SPROC / View / Function level. What happens if you only want a specific schema, a limited number of tables or certain functions? SOL.

F) SQLMetal generates a DataContext file so large that Visual Studio crashes (esp. with Resharper)

Essentially this is an extension of the (E) issue, but it is a very different issue. Where E may result in too many things being generated, the fact that Visual Studio crashes may make exploring the DataContext file impossible. Our DataContext.cs file was over 13MB. The DBML was 4MB.

Solutions:

A) Use a different ORM + CodeSmith

This was not a consideration for us. We love LINQ. We drink the MSFT cool aid, and we weren’t about to move to NHibernate or LLBGEN (though I had a history w/ Hibernate). We have too much invested in LINQ. ( I suspect many are in this camp ).

B) Hand code the DBML

Not an option for us, as there we had a very green OOP / ORM team. We needed something convenient that could work with several very large DB’s with a very diverse set of objects (functions, SPROCs, Tables, Views).

C) Move to Entity FrameWork

This doesn’t provide much shelter from the issues outlined above, not to mention that we have a large amount of existing LINQ code. (we’re vested / locked in)

D) Mix the advantages of SQLMetal & SQL Designer

This is the choice we took. But it required the building of a utility tool that provided a mixture of functionality of SQLMetal & SQL Designer. We were in DBML Hell, and thus only DBMLGod could set us free! (tongue in cheek, but I named the tool just that DBMLGod).

DBMLGod ( SQL Designer + SQLMetal + Hand Coded DBML )

The idea was to take the strengths of SQL Designer and that of SQLMetal and merge them into a hybrid that has all the positive qualities but none of the negative. Plus we need the ability to HAND code very complicated SPROC’s that even SQL Designer doesn’t generate correctly. This succeeded, but it does require a us to develop in a certain way.

SQL Designer

Strengths:

1. Handles Complex Sprocs better than SQLMetal (often times)
2. Allows SPROCs return types to be mapped to entities
3. Can add associations that do not exist in the DB / ER
4. Quick additions of new DB objects (See issue #D above)

Primary weakness: SVN Conflicts

Hand generated DBML

Strengths:

1. Full control over what is generated & how

Primary weakness: Must hand code / Must understand the DSL.

SQLMetal

Strengths:

1. Reduces conflicts (on par with Hand generated DBML)
2. Generates required classes

Weaknesses: See above (A-F)

Ideally we should be able to accumulate strengths without suffering weaknesses.

Implementation of DBMLGod

(Hopefully to be a open source project) – requires my companies sign off.)

In a green field project we might be able to avoid weaknesses of SQLMetal, but given real world scenarios of large existing databases with complex SPROC’s this isn’t possible (for us).

Requirements:

1. Support complex SPROCs (use SQL Designer / hand coded DBML) [ Issue A ]
2. Add associations not defined in DB / ER [ Issue B ]
3. Map SPROC return types to Entities [ [ Issue B ]
4. Change generated SQLMetal DataContext code (support inheritance from custom DataContext Object) [ Issue C ]
5. Quick addition of new DB objects (no waiting 15 minutes for regen of DBML / DataContext file) [ Issue D ]
6. Decrease the size of the generated DBML / DataContext [ Issue E ]
7. Generated DataContext must not be unnecessarily large (make things as simple (small) as necessary but no smaller) [ Issue F ]

How do we do that?

And so the idea is to take the SLOW SQLMetal Generation of the DBML (for large DB) and the faster FILTER / MERGE / SQLMetal Code Gen process and make it fast.

DBMLGod does just that. It gets us out of hell by filtering the Generated DBML, merging those into Hand/Designed DBML files (preferring elements in the latter), generating the code and then replacing code elements in the generated DataContext to get the desired result.

Step 1) Gets value of SLQ Metal DBML Gen but allows us to filter resolving issues E & F

Step 2) Leverages value of Hand/SQL Designer to resolve issue A & B

Step 3) Generate the Code

Step 4) Allows us to resolve isssue C & D

How well does this work?

Addition of DB objects has performance on par with SQL Designer (fast)

SVN conflicts are minimized.

Generation of DBML by SQLMetal (SLOW) can be left to END OF DAY process!

Unresolved issues

SQLMetal capitalizes properties on entity clasess and SPROCs. Default behavior of SQL Designer is to *NOT* do this.

Thus you have 2 choices:

1) Change the SQL Designer generated Sprocs/Entities to conform with SQLMetal generation

2) Use SQL Designer & correct/deal with compilation errors as they occur

Can you believe that MSFT uses different strategies depending on the tool? (UGH, SIGH, HORROR)

But thanks for LINQ! Love it!

Related Post: Unfortunate differences between SQL Metal and SQL Designer

alan.huffman Architecture, C#, Development Infrastructure, LINQ .NET, Coding Practices, LINQ, Microsoft, ORM

We have built a brown field project, which is to say that it has some ugly architectural and coding attributes.

There are 2 mains issues with it.

1. Low code coverage (unit tests)
2. In many cases we implemented the Anemic Domain Model Anti-Pattern (see Fowler’s blog)

The issue, in part, is that LINQ psychologically pushed us in that direction (I think – or maybe I’m just making excuses).

This is an example, but we mapped out an architecture a few years ago that was along the lines of this simple tiered structure:

 

** Note the use of “tbl” for tables & Entity objects is merely to reinforce their connection to the DB due to the 1 table –> 1 object mapping of LINQ. (I don’t use “tbl” in my tables)

All seemed well in the world. We happily went on our way creating partial classes for our tables.

First User Story: Determine the Order cost.

The solution seemed easy enough, loop through the order lines, add up each order line cost, and take the sum.

public Currency Cost( ){

    return this.tblOrderLines.Sum( ol => ol.Cost );

}

But then came the question of where to put this. Obviously OOP would push us to put .Cost() on the tblOrder object. But we faced two problems:

1) By naming the persistence layer as DAL Layer, psychologically we were averse to putting too much business logic there.

Perhaps the DAL layer *is* the Domain Object layer and by merely renaming the DAL Layer as Domain Layer, we would have felt better about it.

2) The second was that the tblOrder object didn’t have enough information to determine the total cost of the order. We had to add taxes, and the amount of taxes changed depending on the state and the type of the order_line.

The Order really needed access to DataContext in order to grab additional information from a tblTaxState table. This is were we made the mistake of putting the COST() into the OrderService

public Currency Cost(tblOrder ord, DBcontext db){

       tblTaxState st = db.tblTaxStates.Where(

                                 txSt => txSt.State == ord.Patient.State

                               ).First();

       Currency cost = new Currency();

        ForEach( var ol in ord.tblOrderLines ){

                    cost += ol.Cost + txSt.ApplyTax( ol );

        }

       return cost;

}

There are a number of ways we could have better solved this. We could have passed the tblTaxState to the tblOrder.Cost() method.

  tblTaxState st = db.tblTaxStates.Where(

                                 txSt => txSt.State == ord.Patient.State

                               ).First();

Currency cost = ord.Cost( st);

Or perhaps even better, would have been to merely pass the DBContext to the tblOrder.Cost() method

Currency cost = ord.Cost(db);

My question is whether that passing around of the DB Context to Domain / Entity objects *SMELLS*.

alan.huffman Architecture, Uncategorized anti-patterns, Architecture, C#, Design Patterns