Pain Alignment - Align the pain with who you want to fix the problem.

As a  manager, I have to build systems to make people do the right thing. For example, if I want software engineers to make the software testable, I make the same software engineers in charge of testing (This is not a majority Microsoft opinion)! I call this strategy Pain Alignment. I heard a great example of pain alignment  on my run this morning and I'm going to share it Verbatim:



   Rocky (not his real name) lived in a house with single paned glass in the city. Being in the city his house was very close to the neighbors.  Unfortunately for Rocky his neighbor's house had nice double paned windows and if the dogs barked outside it didn't bother them.  One day his neighbors decided they needed some yippy dogs.  At first this was fine since the dogs lived in the house, and the neighbors walked the dogs regularly. Soon the neighbors realized walking the dogs was a lot of work, and they installed a dog door. Now the dogs could run into the yard and bark.  There's a lot of activity in the yards in a city, so this was a stressful environment for the dogs,  which resulted in lots of barking - barking which didn't bother the neighbors.



The dogs barked and barked, all day and all night. This was a real problem for Rocky since the dogs woke him up, and then woke his kids up,  which was not cool.  Rocky complained, but the neighbors weren't that interested. To up the pain,  if the dogs woke Rocky up at 2am, Rocky would call the neighbors at 2am to make sure the neighbors felt the pain of the dogs barking.



At first his neighbors tried to fix the problem. First the neighbors tried "dog barks, citrus spray" collars. This worked until the dogs realized if they barked enough the citrus reservoir would become empty and they could then bark as they pleased.  After that the neighbors tried  "dog barks, electric shock" collars.  This was effective till the dogs realized they could weasle out of the collars.  At this point, Rocky was tired of calling his neighbors at 4am, and decided to take a few stabs at his own solutions.



First Rocky tried getting a magic wand from an infomercial which would emit a noise painful to dogs when it heard barks (while being silent to humans). This was completely ineffective, and I include it only for comic relief.



After that Rocky thought hard, he needed a way to increase the pain of the barking to the neighbors so they would deal with the barking problem.  Rocky realized if the dogs pooed every time they barked, this would be painful for the neighbors, so he came up with the following idea. When the dogs bark that applies pressure to their intestine, by coupling the constant barking with some mild laxatives the dogs would poo every time they barked!



Rocky purchased snausages and drilled out 1/2 the sausage snack, and inserted a bit of laxative - the dogs loved eating the snausages.  Now when the dogs barked, they'd poo! This resulted in much pain for the neighbors and within 2 weeks they connected their dog door to a newly built dog run with high fences. The fences blocked out the urban environment which was stressing the dogs, and thus they stopped barking almost immediately.



Remember, if you want to fix a problem align the pain of the problem with the people that caused it.

Why is the internet covered in ads (aka How much does a website makes on ads)

More and more, I've been noticing the internet is covered in ads. While this is annoying it's parallel to the real world. Drive down any physical highway and you are assaulted with billboard ads, the information super highway (does that date me?) is just as covered.

Why is the internet covered in ads? Because it's good money. Lets work through an example, first some terms and values:

TLA	Term	Value	Definition
CTR	Click Through Rate	0.3%	% of times an add will be clicked
EPC	Earning Per Click	$0.10	Average $ received when a user clicks
RPM	Revenue/Thousand Impressions	$3.00	Amount of money you make when you show 1000 ads.

If you can show 1000 ads your website will make 3$.   This doesn't sound like much, but lets look at how much traffic some websites get.



Lets use livestrong.com, a health and wellness website, as an example.   Sites which tell you how much traffic a site gets are compete, alexa and google ad planner.  Compete estimates the livestrong website traffic is 6M visitors/month.   Using this figure and our above RPM, we compute a  monthly revenue of $18,000 or 216,000/year for livestrong.com all without asking their  "eyeball" customers for any money.



And this dear friends is why the internet is covered in ads. 




If you're interested in how money is made in the web, leave comments and I'll do more posts on the topic!

Real World Artificial Intelligence

Today's  AI interaction was a spam chat bot.  Spam is an excellent example of big numbers times times tiny probabilities.



Below is the transcript of me talking to an AI (calikelya3) I especially like lines 5/6 and 7/8.  Impressively there was some pattern matching to handle line 9.





1) me: who are you?



2) calikeyla3: hey Im ashley. Dont you remember?



3) me: Did we meet at that conference?



4) calikeyla3: hey whats up? 22/F here. you?



5) me: are you a computer



6) calikeyla3: hmm. have we chatted before?



7) me: I have a fish



8) calikeyla3: oh ok. i wasnt sure. anyways.. whats up?



9) me: Do you have naked pictures?



10) calikeyla3: No sorry not on this computer



11) me: How do you want to get my money then?



12) calikeyla3:

 im like so boreddddddddd.... theres nothing to do

ohhh wait! i got a idea. have you ever watched a girl strip on a cam before?



13) me: This is what I was waiting for



14) calikeyla3: it turns me on when guys watch me strip on cam. do you want to watch me?



15) me: Yes, how do I give my credit card



16) calikeyla3: yeah? ok well you have to sign up on this site my cam is linked through so i cant be recorded.  it only takes a minute and its free ok?

Dealing with task overload

All of us get to a point where our boss (*) gives us more work than we can handle.   With more work than we can handle one of three types of conversations will occur:

Type	Time	Conversation
B: too Busy	When task assigned	I’ve got too much on my plate, what can we take off to make this fit?
F: I Failed/We're F@#$'d	When task becomes a failure	I couldn’t get this done, now we’re screwed what would you like to do.
Q: I Quit	When burned out	I haven’t seen my family in weeks, my fish died of starvation - I quit!

Even if you're not yet in task overload think very hard about which conversation your boss would rather have.   If your boss would rather conversation F or Q, you need to find a new job(+).



I'll assume after doing the previous mental exercise you're not out looking for work, so let's ponder how to have conversation B. Clearly your boss didn't give you a task because she was bored, she needed the work done.  Bring your boss your current prioritized, costed, task list, and try to fit the new work into it, recommending where the new item should fit in the task list.



At this point, your boss will agree with your recommendation or work with you to change your prioritization to keep you under a realistic work load.



ps. To my surprise some people I respect disagree with my approach and think I'm being naive. They claim that having conversation B shows you are weak, and will put you in debt with your boss.  If you think conversation B is not the way to go, put your thoughts in the comments.

Notes:

(*) Depending on your circumstance boss may be a euphemism for:  partner, wife, husband, significant other, parent etc.

(+) Depending on your circumstance job may be a euphemism for:  relationship.

Do you have a problem?

Recently I became a developer lead.  In case your company uses different titles, a developer lead is a software developer who has software developers reporting to her.

Since I became a lead, I've realized the different roles in an organization see the world differently. The table summarizes how different roles see a problem.

Role	Says	Thinks
Developer	We have a Problem	F@!#$ one more thing to fix, how am I going to get this done
Lead	We have a challenge	Something isn’t the way we want, we’ll get it fixed
Manager (Lead of Leads)	We have an opportunity	My competent teams will solve this.
Director Of Development (Lead Of Managers)	We have a functioning team	You think that's a problem? Ha ha, you'll fix it. You should see what i have to fix.

I'm not sure if this is brilliant, or just funny. Maybe it'll become the career choice acid test :)

AlgoRhyme

Today, I had lunch with the mother of the internet. Among other things she told me her poem that accompanied the spanning tree protocol . Perhaps if a poem was required for each algorithom and protocol submission,  computer science would be simple and clear. With great pleasure I reproduce Radia's poem:

AlgoRhyme



I think that I shall never see
a graph more lovely than a tree.
A tree whose crucial property
is loop-free connectivity.
A tree that must be sure to span
so packet can reach every LAN.
First, the root must be selected.
By ID, it is elected.
Least-cost paths from root are traced.
In the tree, these paths are placed.
A mesh is made by folks like me,
then bridges find a spanning tree.



Radia Perlman

Protect your pocket electronics - ZAGG InvisibleShield screen protectors

ZAGG isn't paying me for this, I'm just trying to save my readers a busted device.




I'm extremely hard on stuff, especially my pocket electronics.  My pockets are jammed with a camera, keys, cell phone, knife, and money clip.  This environment normally destroys LCD screens, but  I found the antidote.  I use ZAGG InvisibleShield screen protectors. These protectors cost 10-20$ per device, are pretty much invisible and protect the screens of every piece of electronics I put in my pocket.



On top of having a great product, ZAGG has awesome customer support. First I incorrectly entered my shipping address, and ZAGG resent my order to the correct address for free. Second, and likely  relevant to more people :) , my screen protector got damaged, and ZAGG sent me another one for free (their normal warranty terms).   Great product, great company, if you stick a device in your pocket I highly recommend ZAGG.

How to query windows azure diagnostics for a time range?

Windows Azure Diagnostics stores data in Azure Table. Unfortunately, Azure Table only supports a single index, which means queries that don't use the index are dog slow.  In azure terminology, that index is the partition key, and Windows Azure Diagnostics sets the partition key on your behalf using the following formula:     0 + DateTime.Ticks

You can generate the Partition key query in powershell for all logs after 7/15 by doing:

PS C:\> "PartionKey gt '0$((Get-Date '2011/7/15').Ticks)'"
PartionKey gt '0634462848000000000'

For the curious, gt is the greater than operator in Azure Table, which is based on ADO.Net DataServices, which is now called WCF Data Services, which is  OData.

F# and the immediate window

(This post is inspired from this Stack Overflow question)



F# debugger integration is quite good, except for the immediate window. The immediate window requires C# style syntax, so calling F# functions is not intuitive. Below are recipes for the immediate window integrated into a console application. As always code can be found here:

// Learn how to use the immediate window for F#.
module Program=

    // To build, add  a reference to Microsoft.CSharp. This is required for dynamic support in
    // the debugger.
    let allowDynamicToWorkFromImmediateWindowByLoadingCSharpBinders = Microsoft.CSharp.RuntimeBinder.CSharpBinderFlags.BinaryOperationLogical

    let plusOne = (+)1
    let xPlusOnes = [1;2;3] |> Seq.map plusOne
    
    []
    let main (args) = 
       printfn  "Try the following in the immediate window:
// plusOne 4
((dynamic)plusOne).Invoke(4)

// Seq.Head xPlusOnes
Microsoft.FSharp.Collections.SeqModule.Head(xPlusOnes)

// let xPlusTwos= xPlusOnes |> Seq.map plusOne 
// Seq.Head xPlusTwos
var xPlusTwos = Microsoft.FSharp.Collections.SeqModule.Map(plusOne,xPlusOnes);
Microsoft.FSharp.Collections.SeqModule.Head(xPlusTwos)"

       System.Diagnostics.Debugger.Break()
       let returnValue = 1 
       returnValue

Using Ads to get free DropBox storage space.

(I did not "invent" this idea, someone told me about it. I figure you'll be interested in the experience, it was rather surprising for me.)

Google makes money on ads, oodles and oodles of it. Up until today, I never "understood" the value of ads.  Today, I got a chance to use ads, here is my story:

Dropbox is a storage service, if your friend signs up by clicking your referral URL, both you and your friend get 250MB of free storage up to 8GB of storage. This means if you can get 32 (8GB/250MB) friends to sign up you get lots of free storage.

As you can imagine, I don't have 32 friends, so the question is how do I get people to click on my URL? Ads!  I signed up for a Google AdWords account, and created an add which read - "Download DropBox and get free 250MB of storage". Here are the stats from my experiment:

• Total Cost: $15.00
• Cost/GB:$1.87/GB
• Impressions (number of times add shown): 3.6K
• Clicks: 257
• Average Cost per Click 0.06
• Conversions (times people installed DropBox after clicking) : 32

In this experiment each of impressions -> clicks -> conversions, is a 10x decrease, which seems astronomically high to me.  Seriously, 1 in a 100 people who saw my ad signed up for DropBox.  No wonder eyeballs are so valuable!

A final note for the clever among you, would it have been cheaper to purchase storage from dropbox instead of paying for ads?  DropBox pricing is $2.40/GB/Year,  and worse you can only purchase it in increments of 50GB.  Thus, using ads and the referral system is a good deal.

Using WinDBG/CDB and SOS to find a thread given a ManagedThreadId

CDB and SOS have more power than most people realize, I’ll use this post to show some some handy cdb/sos tricks while at the same answering the simple question of which thread a ManagedThreadId maps to.

At the end of this post you’ll be able to:

• Map a ManagedThreadId to the correct thread ordinal
• Use CDB to convert from decimal to hex
• Pipe CDB commands through pipeline commands like findstr

Full source for the simple app below can be found here. In the below app we want to figure which thread we’re running on so we can inspect the stack.

    internal class Program
    {
        private static void SleepForever() { Thread.Sleep(Int32.MaxValue); }

        static void Main(string[] args)
        {
            const int countThreadsToStart = 20;
            foreach (var unused in Enumerable.Range(0, countThreadsToStart))
            {
                // Instantiate thread that prints and stores its ManagedThreadId.
                new Thread(() =>
                {
                    // store ThreadId as an object as this forces boxing and lets us find the object on the stack via !dso in SOS.
                    object boxedThreadId = Thread.CurrentThread.ManagedThreadId;
                    Console.WriteLine("Find my threadId of {0}", boxedThreadId);
                    SleepForever();
                }).Start();
            }
            SleepForever();
        }
    }

Step 1, run this app in CDB:

 PS C:\hgs\ig2600_blog\FindTheThread\bin> C:\bin_drop\public_dbg_64\cdb .\Debug\FindTheThread.exe

Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: .\Debug\FindTheThread.exe
 
Find my threadId of 3
Find my threadId of 4
Find my threadId of 5
Find my threadId of 6
Find my threadId of 7
Find my threadId of 8
Find my threadId of 9
Find my threadId of 10
Find my threadId of 11
Find my threadId of 12
Find my threadId of 13
Find my threadId of 14
Find my threadId of 15
Find my threadId of 16
Find my threadId of 17
Find my threadId of 18
Find my threadId of 19
Find my threadId of 20
Find my threadId of 21
Find my threadId of 22
(e28.c94): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
00000000`7744ef90 cc              int     3
0:025> .loadby sos clr

Lets say our goal is to debug what’s running on thread 21, how do we know which thread ordinal we should debug on?   Luckily we have a !Threads command:

 0:025> !threads
ThreadCount:      22
UnstartedThread:  0
BackgroundThread: 1
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                           PreEmptive                                                   Lock
       ID  OSID        ThreadOBJ     State GC       GC Alloc Context                  Domain           Count APT Excepti
on
   0    1  224c 000000000054dd80   200a020 Enabled  0000000002ac90b0:0000000002ac9fd0 00000000004ee680     0 MTA 2    2   c8c 0000000000553bd0      b220 Enabled  0000000000000000:0000000000000000 00000000004ee680     0 MTA (Finalizer)
   4    3  1670 0000000000591620   200b020 Enabled  0000000002ac3be0:0000000002ac3fd0 00000000004ee680     0 MTA
   5    4  1c84 0000000000578370   200b020 Enabled  0000000002ac4160:0000000002ac5fd0 00000000004ee680     0 MTA
   6    5  1a54 000000000057a190   200b020 Enabled  0000000002ac6160:0000000002ac7fd0 00000000004ee680     0 MTA
   7    6  2364 0000000000591f90   200b020 Enabled  0000000002aca160:0000000002acbfd0 00000000004ee680     0 MTA
   8    7   b38 0000000000593570   200b020 Enabled  0000000002acc160:0000000002acdfd0 00000000004ee680     0 MTA
   9    8  1c88 0000000000594df0   200b020 Enabled  0000000002ace160:0000000002acffd0 00000000004ee680     0 MTA
  10    9   ef8 00000000005966b0   200b020 Enabled  0000000002ad0160:0000000002ad1fd0 00000000004ee680     0 MTA
  11    a  21d0 0000000000597f70   200b020 Enabled  0000000002ad2160:0000000002ad3fd0 00000000004ee680     0 MTA
  12    b  22a8 0000000000599830   200b020 Enabled  0000000002ad4160:0000000002ad5fd0 00000000004ee680     0 MTA
  13    c   600 000000000059b0f0   200b020 Enabled  0000000002ad6160:0000000002ad7fd0 00000000004ee680     0 MTA
  14    d  2388 000000000059c9b0   200b020 Enabled  0000000002ad8160:0000000002ad9fd0 00000000004ee680     0 MTA
  15    e  19b8 000000000059e270   200b020 Enabled  0000000002ada160:0000000002adbfd0 00000000004ee680     0 MTA
  16    f  1e9c 000000000059fb30   200b020 Enabled  0000000002adc160:0000000002addfd0 00000000004ee680     0 MTA
  17   10  1c9c 00000000005a13f0   200b020 Enabled  0000000002ade160:0000000002adffd0 00000000004ee680     0 MTA
  18   11  20c4 00000000005a2d40   200b020 Enabled  0000000002ae0160:0000000002ae1fd0 00000000004ee680     0 MTA
  19   12  14bc 00000000005a4600   200b020 Enabled  0000000002ae2160:0000000002ae3fd0 00000000004ee680     0 MTA
  20   13  1994 00000000005a5ec0   200b020 Enabled  0000000002ae4160:0000000002ae5fd0 00000000004ee680     0 MTA
  21   14  19bc 00000000005a7780   200b020 Enabled  0000000002ae6160:0000000002ae7fd0 00000000004ee680     0 MTA
  22   15   1ac 00000000005a9040   200b020 Enabled  0000000002ae8160:0000000002ae9fd0 00000000004ee680     0 MTA
  23   16  2084 00000000005aa900   200b020 Enabled  0000000002aea160:0000000002aebfd0 00000000004ee680     0 MTA
0:025>

In this output, the first column is the thread ordinal (which is what the ~ operator works on).  The second column, is the OSID which is the ManagedThreadID,  but is inconveniently the value is displayed in hexdecimal.  To continue we need to find the thread ordinal, given an OSID of 21 in base 16. To convert 21 to hex, use the evaluation operator (?), and pass in an explicitly base 10 number by pre-pending 0n:

 0:025> ? 0n21
Evaluate expression: 21 = 00000000`00000015
0:025>

Now, re-run !threads, but pipe the output through findstr 15 . Findstr can be run via the .shell command:

 0:025> .shell -ci "!Threads" findstr 15"
  15    e  19b8 000000000059e270   200b020 Enabled  0000000002ada160:0000000002adbfd0 00000000004ee680     0 MTA
  22   15   1ac 00000000005a9040   200b020 Enabled  0000000002ae8160:0000000002ae9fd0 00000000004ee680     0 MTA

OSID 15, is thread 22, lets switch to it with the ~s operator:

0:025> ~22s
ntdll!NtDelayExecution+0xa:
00000000`7745009a c3              ret

To verify we have the correct thread, we can dump the stack and inspect the boxedThreadId Object. We do this with !dso to (dump stack objects) and then run !do (dump object) on the int32.

0:022> !dso
OS Thread Id: 0x1ac (22)
RSP/REG          Object           Name
0000000021F7E4C0 0000000002ac2110 Microsoft.Win32.SafeHandles.SafeFileHandle
0000000021F7E590 0000000002ac3048 System.IO.StreamWriter
0000000021F7E5C0 0000000002ac3318 System.Byte[]
0000000021F7E5E0 0000000002ac8f60 System.Threading.ExecutionContext
0000000021F7E638 0000000002ae7fe8 System.Int32
0000000021F7E640 0000000002ac2070 System.String    Find my threadId of {0}
0000000021F7E680 0000000002ac1fe8 System.Threading.ContextCallback
0000000021F7E698 0000000002ac8f60 System.Threading.ExecutionContext
0000000021F7E6A0 0000000002ac8ef8 System.Threading.ThreadHelper
0000000021F7E6B0 0000000002ac8f60 System.Threading.ExecutionContext
0000000021F7E6C0 0000000002ac3490 System.IO.TextWriter+SyncTextWriter
0000000021F7E6D0 0000000002ac8f60 System.Threading.ExecutionContext
0000000021F7E6D8 0000000002ac8ef8 System.Threading.ThreadHelper
0000000021F7E6E0 0000000002ae7fe8 System.Int32
0000000021F7E6F0 0000000002ac8ea0 System.Threading.Thread
0000000021F7E700 0000000002ac8f60 System.Threading.ExecutionContext
0000000021F7E748 0000000002ac1fe8 System.Threading.ContextCallback
0000000021F7E750 0000000002ac8f60 System.Threading.ExecutionContext
0000000021F7E760 0000000002ac8ef8 System.Threading.ThreadHelper
0000000021F7E770 0000000002ac8ef8 System.Threading.ThreadHelper
0000000021F7E7A0 0000000002ac8f60 System.Threading.ExecutionContext
0000000021F7E7B0 0000000002ac8ef8 System.Threading.ThreadHelper
0000000021F7E7C0 0000000002ac8ef8 System.Threading.ThreadHelper
0000000021F7E810 0000000002ac8f20 System.Threading.ThreadStart
0000000021F7E978 0000000002ac8f20 System.Threading.ThreadStart
0000000021F7EBB0 0000000002ac8f20 System.Threading.ThreadStart
0000000021F7EBC8 0000000002ac8f20 System.Threading.ThreadStart

0:022>!do 0000000002ae7fe8
Name:        System.Int32
MethodTable: 000007fee50bc848
EEClass:     000007fee4c40890
Size:        24(0x18) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee50bc848  400047b        8         System.Int32  1 instance               21 m_value
0:022>

Note the value is 21, exactly what we were looking for! Leave a comment if you have things you’d like to see me debug.

Use Paint.Net to Highlight key areas in an image to get your point across

(Since posting this I've also become very fond of Jing, which is the tool I use when taking screen shots)



Earlier, I mentioned how important communication is to software engineering. A key tool in communication is being able to highlight portions of images.  For this task I use some basic features of Paint.Net.  In the below image I want to draw your attention to the name of the dress, and the name of the color of the dress. Without these highlighted regions, it's likely your eye wouldn't have picked up the points I wanted you to see.

To perform these highlights in Paint.Net do the following:

• Copy image into Paint.Net


• Select the ellipse tool


• Specify Fill Options as Draw shape with outline


• Set Secondary color(the fill color) to yellow


• Set transparency to 50 %


• Draw ellipse

Cool Tools: VirtuaWin

Virtual desktop software allow you have several desktops, and switch between them easily.  This is the equivalent of virtual memory, but for desktop real estate. Good virtual desktop software allows you to move windows between virtual desktops, and will have good usability including hot keys.  VirtuaWin is a great free virtual desktop app for windows,  go install it .  Once installed you should do the following:

•     Customize the notification area to always show the VirtuaWin Tray Icon


•     Use C-A-Right, C-A-Left, C-A-Up, C-A-Down to navigate between your desktops


•     Left Click on the VirtuaWin icon to interact with your windows


•     Right Click on the VirtuaWin icon to play with settings

Enjoy, and be sure leave comments if there are tools you'd like more information on.

Cool Tools: Paint.Net

Paint.Net is a nice compromise between advanced picture editing software, and picture editing software usable by humans. The biggest problem I had with paint.net is figuring out how to download it. In what I assume is an attempt to make advertising dollars, this website exists that make it impossible to download Paint.net without accidentally downloading crap.  Here's a direct paint.net download link.



Update here are my usages of Paint.Net:

• Highlighting portions of an image 


• Other you'd like to see

Cool IEnumerable Tricks Part 1: GroupBy

IEnumerable, or what most people refer to as LINQ, has changed the way I program. If you love LINQ and you already fully understand this blog post, leave a comment with a topic you'd like me to post about. If you're not so comfortable with LINQ, try to follow along. This post will only take a few minutes, but may change the way you program for life.

While we go through this blog post (and future posts in this series) make sure you understand how every function works, if you don't understand a function, you certainly won't understand the next function. Some of this stuff is hard to grasp and I find the best way to understand the code is to try it in a debugger. It took me a few tries to wrap my head around what is going on, so expect to spend some time experimenting with this code.

I learn best when I'm trying to solve a problem, so lets define a simple problem: How to determine if the random number generator equally distributes integers mod 10.

Step 1: Generate a stream of random numbers:

var seed = new Random();
var countSamples = 10*1000
var randomStream = Enumerable.Range(0,countSamples).Select(r=>seed.Next());

In the above sample, we generate countSamples random numbers. We do this by producing countSamples integers via Enumerables.Range, than for each of those integers we apply a transformation. In this case the transformation ignores the input value, and returns a random number.

Step 2: Compute how many of each random number have each modulus (0..9).

randomStream.GroupBy(r=>r%10).Select(x=>new {x.Key,count=x.Count()}).OrderBy(r=>r.Key);

In the above sample, we use GroupBy to split the random numbers into groups based on their %10 value. For each group, we return a new object containing the modulus (x.Key) and the count of each group. Finally we sort based on the modulus.  The output is below:

Key	count
0	959
1	938
2	1037
3	1028
4	951
5	1009
6	1029
7	1011
8	1038
9	1000

In a future post I'll describe how to create our own IEnumerable functions.

Leading by praise: How to use a carrot when you really want a stick.

Many naive leaders believe they have two ways to enact change in their followers.  The first way is "the carrot" aka praising good behavior. The second way is "the stick"  aka criticizing bad behavior.  In my experience, the stick is extremely ineffective. I'd say the only time the stick has any positive impact is when the follower doesn't realize she's doing something wrong, or how serious that wrong thing is.  

The goal of this post is how to use a carrot when you want a stick.  First lets recall how to use a carrot effectively in general: Keep it timely, short, specific and often (the one minute manager is my hero).  Any time you see a new good behavior praise it.  When you praise,  be as specific as possible so the follower knows both the behavior you are praising and the valuable impact of that behavior.  It's easy to praise when things go well, but how do we praise when things are going poorly?

When things are going poorly, there's only one thing you can do - put your followers in a position where they perform good behaviors.  Scope their work so they have no choice but to perform good behaviors.  From that good behavior, apply praise.  Even if that behavior is accidental, still give praise.   Slowly, sometimes painfully slowly, you'll get more opportunities to give praise, and you'll get more good behaviors.  Keep giving praise, eventually the fruit of your labor will pay off.

Praising weak performers can be a time consuming delicate process, but it's like lighting a fire: at first you have to work really hard to spark the tinder, then pay close attention while the kindling catches to make sure you don't lose your delicate flame, finally you'll have a fire raging, and you can lean back and relax knowing not only have you lit a fire, but that fire can be used to light other fires.

Google killed the video store

As a younger man I remember arguing with a co-worker about the death of local storage. 



Me: Who wants to store data themselves?  I can store all data in the cloud, where it will be preserved forever!

Wiser co-worker: What incentive does this cloud have to keep your data? I'll keep my data on my local hard drive thank you very much!

 

8 years later my co-worker was  correct. I got the following letter from Google today:



Later this month, hosted video content on Google Video will no longer be available for playback. <snip> We’ve added a Download button to the video status page, so you can download any video content you want to save. If you don’t want to download your content, you don’t need to do anything. (The Download feature will be disabled after May 13, 2011)

Ouch for being wrong,  and double ouch for having to download all my videos to my hard drive.

Notes:

• Astute readers will note, I actually got a decent deal since I got 8 years of free storage and $/GB has plummeted in that time.


• This isn’t the first time my “bet on a cloud service” has failed. I lost all my face tags when polar rose was bought by apple and shutdown.


• I’m a slow learner, and still use the cloud for the primary copy of all my media.

Software Engineering “The Soft Skills”

To date, I've focused my blog on the "computer" parts of software engineering. However, as I've grown as a software engineer, one of my largest aha moments is the realization that software engineering is all about people. As such, I'll start posting articles dealing not just with computers, but also with communication, people leadership and management. These "people skills" are as important to a software engineer as the ability to crank out a design specification. For the hardcore among you that think people skills have nothing to do with software, consider this.

Impedance MisMatch: IEnumerable<T> to DataTable

I had to work with a class that consumed a System.DataTable today. DataTable is an old class that pre-dates generics in the CLR. DataTable carries typed data as an array of System.Object - double plus yuck! As you can imagine the code I use these days has compiler verified type safety by using IEnumerable everywhere. So how did I convert from my beloved IEnumerable to the yucky DataTable? With reflection of course! This simple method solves our problems:

public static DataTable ToDataTable(IEnumerable rows)
{
    var tableToReturn = new DataTable();

    // get properties of T
    var properties = typeof (T).GetProperties().Where(p => p.MemberType == MemberTypes.Property);

    // Convert T's properties to columns in the DataTable
    foreach (var p in properties)
    {
        tableToReturn.Columns.Add(p.Name, p.PropertyType);
    }

    //populate rows
    foreach (var row in rows)
    {
        T row1 = row;
        var propertyValuesAsEnumerable = properties.Select(p => p.GetValue(row1, null));
        tableToReturn.Rows.Add(propertyValuesAsEnumerable.ToArray());
    }
    return tableToReturn;
}

Powershell script to enable windows to capture localhost traffic in wireshark

If you want to understand why the following scripts work read this post. Otherwise just paste the following into an elevated powershell window:

Setup windows networking to allow localhost capturing in wireshark:

# Find the network configuration that has the default gateway.
$defaultAdapter = Get-WMIObject Win32_NetworkAdapterConfiguration | ? {$_.DefaultIPGateway}
if (@($defaultAdapter).Length -ne 1) {throw "You don't have 1 default gateway, your network configuration is not supported" } 
# Route local IP address via the default gateway
route add $defaultAdapter.IPAddress[0] $defaultAdapter.DefaultIPGateway
Write-Host "Start capturing on localhost by connecting to $($defaultAdapter.IPAddress[0])" 

Return windows networking to normal configuration:

# Find the network configuration that has the default gateway.
$defaultAdapter = Get-WMIObject Win32_NetworkAdapterConfiguration | ? {$_.DefaultIPGateway}
if (@($defaultAdapter).Length -ne 1) {throw "You don't have 1 default gateway, your network configuration is not supported" } 

# Stop routing localhost traffic to the router.
route delete $defaultAdapter.IPAddress[0] 

Remember, you won’t see traffic to localhost (127.0.0.1) but traffic to your network adapter’s IP address as listed in the script.

How did we get a 53 byte packet size in ATM?

I'll be honest, I don't know squat about ATM, but I was having lunch with this fellow, and he told me the story of the 53 byte ATM packet. 

You can find more details on Wikipedia, but here’s the synopsis:

(Disclaimer: I’m not an expert in ATM; nor am I trying to teach you technical details about ATM networks; so I’ll hand wave and trade off accuracy for simplicity. For example, ATM does have variable sized packets which it divides into cells, and it is the cells which are 53 bytes long. However, since the closest thing to a cell in common networks is an Ethernet packet, I’ll simply refer to cells as packets.)

ATM is designed to be shared between data network applications, and voice network applications(+).

In data networks we want large packets because this gives maximum efficiency.  This is because each packet has a fixed size header and thus the more data you can transmit per packet , the higher your ‘real’ throughput.

For voice networks we want to reduce latency.  Latency is the fancy word for delay.

If voice packets are delayed, you’ll hear them at the receiver as an echo; this is un-acceptable and thus to mitigate delays you’d need to install an echo canceller (++).
In the USA, the country was big enough (+++) that no matter how small the packet size was chosen to be, you’d always need echo cancellers – therefore America proposed an optimal packet size of 64 bytes.

In France, which was much smaller then America, if the packet size was chosen to be 32 bytes the packet could get from one end of the country to the other without enough delay to require echo cancellers. Thus France pushed for a 32 byte packet size.

To achieve consensus, a compromise was required and a 48 byte payload size was chosen.  This did not meet France’s need, nor was it optimal for the US; however, this is what happens when you require consensus.
But wait you ask, where do the last 5 bytes come from? Ah, the committee was very concerned about the overhead of the packet header, so the committee decided the packet header could not exceed 10% of the packet size, thus the committee chose a 5 byte header.  48 + 5 == the 53 byte ATM packet size; which is the optimal solution for no one.

Notes:
(+) At the time ATM was designed, people made long distance calls over voice networks.
(++) If you’re old enough to remember the early days of skype, you needed to use headphones otherwise the echo was unbearable. Today computers are powerful enough to do echo cancellation without blinking.
(+++) Compute big enough by calculating time required for light to traverse the USA.

The cloud lets you evaluate the cost of performance optimizations

One of the things I love about cloud computing is you can put an honest price on computing time.  You can than balance the human engineering time required to optimize code (and often have more complex code) vs just paying for the cloud to do it.  The Zillow rent estimate post speaks to this brilliantly:

We implemented the Rent Zestimation process as a software application taking input from Zillow databases and producing an output table with about 100 million rows.

We deploy this software application into a production environment using Amazon Web Services (AWS) cloud.  The total time to complete a run is four hours using four of Amazon EC2 instances of Extra-Large-High-CPU type.  This type of machine costs $1.16/hr.  Thus, it costs us about $19 to produce 100 million Rent Zestimates which is the same as a 3-D movie ticket or about 5 gallons of gasoline in New York City today.

A few things to note about this quote:

• If your data processing can be done in parallel, you’d have the same cost but at a lower time since you can run all instances in parallel
• If your data processing must be done in serial, and you want results faster than the largest instance can compute them, buy some coffee you’ll be up doing optimizations.
• Gas prices are through the roof in New York

Using LINQ to XML from F# (and a bit of duck typing for fun)

So you want to use XML from F#? here are the tips and tricks I discovered today to clear my roadblocks.

Tip 1: Reference the System.XML assembly from FSI so you play with the REPL. To do this I add the #r in block comments at the top of the file  like so:

(* Press Alt-' to send the below to the interpreter.
#r @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\System.Xml.Linq.dll"
*)
open System.Xml.Linq

Tip 2: F# doesn’t perform implicit conversions. Lets assume you’re translating the following C# code:

var xelements = XElement.Parse("<myXML/>");
var interestingElements = xelements.Descendants("myXML");

The naive translation below doesn’t work because XElement.Decendants doesn’t take a string, it takes an XName. 

//FAILS to compile This expression was expected to have type XName but here has type string 
let interestingElements = xelements.Descendants "myXML"

The reason the above code works in C# is because an implicit conversion operator exists and is called. However in F# implicit conversions are not called, and you have to call them yourself, something like the painful to read:

let interestingElements = xelements.Descendants <| XName.op_Implicit "myXML"; 

Now, it turns out we can use duck typing to make an implicit conversion operator (this is mentioned lots of places on the web but I’ll repeat it for succinctness)

// Create an implicit conversion operator, it's inline which means it will work on any types that support the implicit conversion.
// This is duck typing.
let inline convert_implicit(arg : ^source) : ^target = ((^source or ^target) : (static member op_Implicit : ^source -> ^target) (arg)) 

And finally we can write the code I needed:

let interestingElements = xelements.Descendants <|  convert_implicit "myXML"; 

Supply, demand and the trackball market.

The author of this blog is a devout trackball man, and as any devout trackball man can tell you, it has been trying times for trackball users in the last few years. You see trackballs were never really in style, and no one has made a new trackball for a while.  It's so bad out there that one of my favorite trackballs now sells used for over 200$ and new for 600$.



Crazier than that you can send away to get your trackball reconditioned on ebay, for a whopping 100$, where they'll clean it and put on a new cord.



Thankfully, there has been movement in the trackball market. Logitech released a new wireless trackball!!! So far I love the tiny USB receiver (which fits in the trackball when not in use). I wish the trackball was larger, but after several months of use, I can say it's a perfectly reasonable device.

Decoding Ubuntu Version Numbers

I've always been fond of the Ubuntu code names, "breezy badger" and  "warty warthog" are some of my favorites. I know the code names are in alphabetical order, but I never understood how the Ubuntu folks got the version numbers. Why does 10.10 come after 10.04? Today I realized it's because the version numbers are release dates,  and releases are every April and October. So, this April's release "Natty Narwhal" will be 11.04.  Learn something new every day.  Oh and if you don't know what a  narwhal is,  it's  worth checking out.

If you run firefox– install https everywhere now

I'll do a post later about session hijacking, firesheep and the evils of non https based websites. In the meantime, if you use firefox be sure to the run the https everywhere extension.  Without https everywhere I would be concerned about accessing my accounts on a public wifi network.  HTTPS Everywhere forces the browser to use https when accessing any website that supports both http and https.

(By the way If you have a security question, or question on any of the topics I cover on my blog leave a comment and I’ll be happy to do a post answering your questions)

The 3 A's of security when buying a coffee

The three A's of security are  authorization, authentication and auditing.  While these A's apply to computer security they are also present in most transactions.  For example, lets look at what happens when Igor wants to buy a coffee:



When Igor goes to buy a coffee with his credit card, the cashier starts by asking Igor for a Picture ID, then the cashier swipes Igor's card (which dials the head office to make sure there is money), finally a receipt is

printed. After all that Igor gets his coffee. Let's look at the 3 A's in this transaction

• Authentication - Prove the entity is the entity -  Ensure Igor is Igor by making sure the picture on his drivers license matches his face.


• Authorization - Prove an entity is allowed to do something - Ensure Igor is allowed to place a 5$ charge on his credit card.


• Auditing - Record what has occurred - Write down that a coffee was purchased.

Authentication, Authorization, and Auditing in the coffee shop

Why do I keep getting exception code e0434352?

If you want to know how to debug CLR exceptions using cdb then read this post.



Exception code e0434352 is the exception code used internally by the CLR to represent most exceptions(*).



Regardless of if you throw a System.NullReferenceException or a System.ArgumentException in C#, you'll throw a SEH exception e0434352 under the covers.



A fun way to validate this theory is to watch what happens to the CLR exceptions settings in cdb. Fire up cdb, and see the state of clr exceptions:

0:000> .shell -ci "sx" findstr clr
 clr - CLR exception - second-chance break - not handled
clrn - CLR notification exception - break - handled
.shell: Process exited 

Now, set the exception handler for exception number e0434352 and recheck the value of the clr exception handler:

0:000> sxe e0434352
0:000> .shell -ci "sx" findstr clr
 clr - CLR exception - break - not handled
clrn - CLR notification exception - break - handled
.shell: Process exited 

Armed with this knowledge I expect this post makes more sense.



NitPickers Corner:

(*) I know of at least Divide by Zero not using this exception code.

Using NUnit to explore static field initialization

Given the following code:

class Constants
{
    public static readonly Constants Instance = new Constants();
    public static readonly string ApplicationName = "ig2600";
    public readonly string CachedClassName;

    public Constants()
    {
        CachedClassName = GetQualifiedClassName("Constants");
    }

    public static string GetQualifiedClassName(string className)
    {
        return string.Format("{0}::{1}", ApplicationName, className);
    }
}

Which of the below tests pass?

[TestFixture]
public class TestConstants
{
    [TestFixture]
    public class TestConstants
    {
        [Test]
        public void TestOne()
        {
            Assert.That(Constants.Instance.CachedClassName, Is.EqualTo("ig2600::Constants"));
        }

        [Test]
        public void TestTwo()
        {
            Assert.That(Constants.Instance.CachedClassName, Is.EqualTo("::Constants"));
        }
    }
}

Not sure? Download the code with mercurial from  https://bitbucket.org/idvorkin/ig2600_blog, then fire up VS and NUnit, and look at the StaticFieldMystery project.

Four Visual Studio Extensions You Need To Be Using

I spend a lot of time in Visual Studio and these extensions  make visual studio even more awesome.  If you haven't used VS to install  extensions, it's  trivial.  Follow along by doing:



Tools -> Extension Manager -> Online Gallery -> Search



Here are the extensions  I use, and strongly recommend you try. I have seen no adverse problems from running any of these extensions.  Here are the extensions I use: 



Productivity Power Tools (by Microsoft; free): 

For VS2010: Adds a Solution Navigator,  a better Solution Explorer,  and cool tabs features like pinning and color coding by project. 



For VS2012: I haven't explored yet, but will update





PowerCommmands  Visual Studio (by Microsoft, free): 

For VS2010 Only: Several random features, the ones I really enjoy are Open Commmand Prompt here and Open Containing Folder.



NuGet Package Manager (by Micosoft, free): 

A package manager for .net. Say you want to use an open source library in your VS project, you open package manager and say Install-Package . With that the project you requested along with it's dependancies are downloaded and liked to your project. Must have when working with popular .net projects.





Resharper (by Jetbrains, 199$) This is "the" extension for visual studio. If you haven't used it, get the trial, I suspect you'll conclude it's worth paying for.  Resharper has several strong refactorings, plugins to fix stylecop errors, and first class support for running NUnit. 





VSVim (by Jared Parsons, free)

If you don't love VI you skip this.  If you love VI you get a fifth must use extension.  VsVim gives VI keybindings to visual studio. It's being actively developed and gets better every release. 



Again, I strongly recommend if you use Visual Studio that you try out these extensions right now, trust me, it’s worth the effort.

SSDs rock - but not for the reasons you think

I bit the bullet and picked up an SSD. It's expensive, but it had some surprising benefits. The normally quoted benefit is speed - which I really don't notice. Most operations went from some annoying amount of time, to some other annoying amount of time. I did notice two things:



1) Silence:  I'm so used to hearing my laptop harddrive I couldn't figure out what I was missing - there is no more annoying grindings; clicking or whirring.



2) Coolness: SSD's run cooler - which means less spinning fan, which means more silence



UPDATE: The most valuable thing about the SSD is the harddrive doesn't die when I drop the laptop, if you don't have an SSD in your laptop it's totally worth it!





(Update: A short,  highly entertaining talk about SSDs - mostly for servers here)