<![CDATA[Christoph Rüegg]]> 2012-01-13T14:42:52+01:00 http://christoph.ruegg.name/ Octopress <![CDATA[Lokad.Cloud Architecture Refresh]]> 2011-08-04T13:29:00+02:00 http://christoph.ruegg.name/blog/2011/8/4/lokadcloud-architecture-refresh In a recent post about new deployment and versioning approaches in Lokad.Cloud I mentioned that I’m also heavily refactoring the old cloud service framework and runtime. That refactoring was long due but also required to support these new approaches effectively.

In essence, developing Cloud Services still works as before. There is a framework library (Lokad.Cloud.Services.Framework) that provides base classes for a small set of service types that you can derive from. The following figure shows the dependencies of all involved components:

Previously the framework also contained the complete runtime with AppDomain isolation and all. This is no longer the case (since we want to use the new deployment approach). Instead, the framework now comes with service runners, which are lightweight classes that take already created service instances plus their settings and can be used to run cloud services directly in the current thread, without any isolation. This comes handy for easier debugging and testing. In simple scenarios it might even be good enough for production or integration into another system.

However, in production scenarios you often do want proper isolation, more robustness and some deployment story. That’s where the new AppHost comes in.

Introducing the Lokad.Cloud AppHost

The new deployment approach is currently implemented in a prototype, Lokad.Cloud AppHost. I’ll introduce the AppHost in more details in a later post. Important for now is that it comes with two assemblies, AppHost and AppHost.Framework. AppHost.Framework is essentially a set of contracts, while AppHost implements the actual runtime environment. Both are quite small and simple. The typical architecture anticipated in the prototype is as follows:

  • Your Context
    Represents the whole environment where the AppHost is executed to the AppHost itself. This is why AppHost has no dependencies at all (except SharpZipLib, but that will likely be dropped soon). Thanks to this abstraction, AppHost is completely neutral to where and how it is executed. The context also provides a deployment reader and thus decides where and how application deployments are stored.

  • Your Worker Process
    This would be the process where the whole application is executing, e.g. a Windows Azure WorkerRole, an Windows Service or even some CLI application. The worker builds the “host context”, creates an AppHost Host instances using said context and then starts and stops the host on demand.

  • Your Entry Point
    The entry point of your application that is hosted using the AppHost. The entry point class type is chosen in the deployment itself, and automatically created in one or more runtime cells (again as specified in the deployment), isolated by AppDomain and in its own thread.

Note that this figure does not mention Lokad.Cloud Services, Storage or Provisioning at all. Indeed, AppHost could be used to host all kind of applications (e.g. even some business application based on Lokad.CQRS).

Hosting Lokad.Cloud Services in AppHost

That’s all nice and well, but the primary scenario is to run Lokad.Cloud Services. One of the design targets of Cloud Services have always been simple usage, achieved in parts by tight integration of our storage and provisioning toolkits into the services framework (opinionated on infrastructure). Luckily this gives us the opportunity to fully provide complete AppHost Context and EntryPoint implementations. The complete services solution now looks like this:

Note that the services framework no longer depends on Provisioning, and does not depend on any AppHost infrastructure at all. Neither AppHost nor Provisioning thus leak into your cloud services implementations. The separation between AppContext and AppEntryPoint also reflects that they run in different places: AppContext is used directly in the host process, while AppEntryPoints run in the isolated runtime cell AppDomains. This becomes clear when we visualize the complete solution:

This looks quite complicated and like a lot of infrastructure just to support that little yellow box on the top right. But this is somewhat misleading, as all the components are very focused and most of them small and independent.

A closer look at what is actually deployed on the worker (e.g. your Azure WorkerRole) reveals that there is really nothing more than the AppContext opinionating the AppHost towards Lokad.Cloud Provisioning and Storage and then connecting this context with the AppHost and run it in the worker process:

Similarly, the actual (versioned) application deployments need to contain only the assemblies shown in the following figure. Obviously there are your cloud services, but also the entry point and the service framework:

All these parts are thus movable and “replaceable” per deployment. This brings up some nice opportunities, as you can patch and replace any of these assemblies in specific deployments without worrying about compatibility with the worker process (this used to be an issue in the past). You can change the scheduling, add new cloud service types or even replace the framework and entry point completely with your own code. Technically there’s also no need to keep it separated into three assemblies, but the isolated EntryPoint helps keeping some dependencies like AppHost out of your cloud services.

Overengineered?

I claim it is not. If you do want all of these:

  • Storage:
    Robust storage (especially important for remote cloud storage) that is very easy to use

  • Provisioning:
    Automatically scale your worker instances (cloud scenario) based on demand

  • Deployments:
    Easily switch between deployments, fast, versioned including settings, in Git style.

  • Runtime:
    Robust multi-cell cloud application hosting, self-healing to some degree.

  • Cloud Services:
    Compute agents that are easy to implement.

then you do need all these components. You can either have them all in one huge monolithical assembly and your logic depending on all of them, or you can isolate them logically, keep them simple and focused and avoid unnecessary dependencies, as suggested in the presented architecture.

]]> <![CDATA[Cleaning up after migrating from Hg to Git]]> 2011-07-30T11:35:00+02:00 http://christoph.ruegg.name/blog/2011/7/30/cleaning-up-after-migrating-from-hg-to-git There is a lot of guidance out there on how to migrate from Mercurial to Git, but they often leave you with a repository in a bad state. Even more so if it originally was a subversion repository, then migrated to Mercurial and now finally to Git.

The Lokad.Cloud repository was such a case. The committers and authors in the commit history were a complete mess, but that’s not that much of an issue in practice. Worse is the fact that most text files were stored with CLRF line endings instead of LF internally. Git supports platform-native checkouts (CRLF on Windows, LF on Linux) quite nicely, but it only works well if text files are normalized to LF internally when committed. I strongly recommend doing that, as it will save you from a lot of trouble later on. Luckily it is also the default behavior for new repositories.

Migration: Fast-Export to Git

This is the usual procedure that properly converts branches and tags to the git equivalents:

1
2
3
4
5

git clone git://repo.or.cz/fast-export.git
mkdir git_repo && cd git_repo
git init
/path/to/hg-fast-export.sh -r /path/to/mercurial_repo
git checkout HEAD

Normalize the whole history to LF line-endings

This step is only needed if all or some of the commits have been using non-LF line endings internally. If the repo once was in Subversion on Windows this most certainly is the case, but not necessarily on pure mercurial repositories. You can find out whether this is an issue, if you remove your git index and then reset. If a lot of files are now listed as modified, you better fix it as described here, if not you can skip this step.

1
2

rm .git/index
git reset

I recommend to do this step in Linux as it didn’t work well for me on Windows.

First we need to turn off any automated git end-of-line handling. Unfortunately this is controlled in multiple places (for historical reasons). First there is the core.autocrlf config we need to turn off:

1

git config core.autocrlf false

Then we need to get rid of all the .gitattributes files in your repository in case they specify any automatic eol handling. This is not necessary in most of the cases, yet the repository I was dealing with used to be a hybrid git/mercurial repo some time ago and thus did already have a gitattributes file. If there is one, delete it and commit. Afterwards your current working directory should be clean, since git no longer wants to fix your line endings on any touched text files.

But to make sure the .gitattributes file in previous commits don’t mess with us, we need to drop it in all commits (single line):

1
2

git filter-branch --prune-empty --index-filter
   'git rm --cached --ignore-unmatch .gitattributes' -- --all

After that we finally can go converting all the text files to LF line endings, with another history rewrite (single line):

1
2

git filter-branch -f --prune-empty --tree-filter
   'git ls-files -z | xargs -0 dos2unix --skipbin' -- --all

What this does is for every commit, for all files that are not binary, convert them to LF endings using dos2unix. In my case there are some paths with spaces in them (don’t ask..), so I switched over to NULL-character separation using the -z and -0 options.

To ensure the normalization is enforced in future commits (especially from people forking your repository and then send you pull requests), create a new .gitattributes files containing at least something like * text=auto. The config option core.autocrlf however is not only local but also depreciated. You can remove it completely using

1

git config --unset core.autocrlf

Clean up committers and authors

You can get a quick overview on how badly the authors are off using

1

git shortlog -se

Luckily, fixing them is not that difficult, with yet another history rewrite:

1
2
3
4
5
6
7
8
9
10
11

git filter-branch -f --env-filter '
if [ "$GIT_COMMITTER_NAME" = "bad user name" ]
then
export GIT_COMMITTER_NAME="correct user name"
export GIT_COMMITTER_EMAIL="correct email address"
fi
if [ "$GIT_AUTHOR_NAME" = "bad user name" ]
then<br />export GIT_AUTHOR_NAME="correct user name"
export GIT_AUTHOR_EMAIL="correct email address"
fi
' -- --all

Housekeeping

After all these rewrites it would be a good time to do some git maintenance, i.e.

1

git fsck --full

to check and verify your repository, drop no longer used blobs with

1

git prune

and then clean up and optimize your local repository using

1

git gc --aggressive
]]> <![CDATA[Lokad.Cloud Application Deployment and Versioning Refresh]]> 2011-07-15T17:10:00+02:00 http://christoph.ruegg.name/blog/2011/7/15/lokadcloud-application-deployment-and-versioning-refresh Disclaimer: I’m a major contributor to the Lokad.Cloud opensource project. Lokad.Cloud is a framework for distributed computing in Windows Azure, plus a set of independent toolkits like Lokad.Cloud.Storage for simpler and more reliable cloud storage access and Lokad.Cloud.Provisioning for dynamic worker auto-scaling. We use Lokad.Cloud at Lokad to deal with our massive and rapidly changing computation demands.

At the very beginning of the Lokad.Cloud project we decided to not rely on the Windows Azure management tools to deploy new versions of our application. Instead we implemented a dynamic worker role - initially deployed once using the Windows Azure tools - that provides a runtime environment that can load and unload applications on demand, without even recycling the azure virtual machine. The applications are isolated in a separate AppDomain so we can unload them safely, plus for sandboxing.

This approach worked out nicely for us, both for apps that don’t change for months and apps where we often have multiple redeployments per hour. Yet over the last year we gained a lot of experience and discovered some dark spots in the current stable release. Most of them were “good enough” back then, but are starting to get in the way:

  • Non-atomic deployments
    The deployment mainly consisted of an assembly blob (zip file containing all the assemblies), a config blob (IoC configuration to add application-specific registrations and config like additional connection strings) plus service settings distributed over multiple blobs. Since workers automatically discover new deployments themselves, deployments are not atomic. Some worker could see new assemblies but old and possibly incompatible IoC configuration, causing undefined behavior until it catches up.

  • Lots of blobs to poll for changes
    By design, all workers are completely self-contained and self-healing. A worker is never “contacted” in any way from outside, except via cloud storage (queues, blobs, tables). Since the deployments are non-atomic and in particular service settings spread over multiple blobs, they all have to be polled for change repeatedly on every worker. Polling for changed etags is not expensive but causes latency and can sum up if there are lots of services and worker instances. Even worse, some service settings also contained state (for simplicity) and thus change quite often.

  • Replacing an application is easy, getting back not so
    It is very easy to redeploy, but there’s no way to get back to the previous state unless you have a backup ready or can rebuild it from sources. Upgrading becomes much safer if it is easy to get back, reducing the burden of shorter deployment cycles.

  • Growing demand for stronger runtime
    E.g. to support multiple runtime “cores” or “cells” on each worker with independent scheduling and customizable assignment/affinity.

Hence I started refactoring the Lokad.Cloud service framework recently, including reworking the handling of app deployments (not released yet):

Concentrating service settings to single blob

Cloud services have been refactored so that they no longer have to manage their own settings. Instead all settings are now stored in a single blob. Settings include parameters like whether a service is disabled or the trigger interval for scheduled services. Settings generally change rarely (e.g. manually through the management console) so the new settings blob still changes only rarely and conflicts are no issue (can be handled with optimistic concurrency). This brings the number of blobs to poll for drastically down to three, reducing a lot of unnecessary storage I/O and thus latency.

Separating deployments from currently active deployment

Previously there were just three blobs (assemblies, config and settings) for the currently active deployment. Changes were almost immediately applied on all workers.

From now on we can have multiple deployments exist in parallel. The “currently active deployment” is given by a pointer to the chosen deployment. Deployments are now read-only. If we want to change anything in a deployment (e.g. change some settings) we essentially create a new deployment and update the active deployment pointer to point to that new deployment instead. Let’s call that new pointer HEAD.

Only one blob to poll

Since deployments are read-only, the only blob we have to poll is HEAD. Since no other polling is needed, we can easily poll more often to get much more reactive workers. If we poll once every 15 seconds, we get transaction costs of around US$ 0.17 per month per active worker plus maybe US$ 0.10 for bandwidth (note that most of the time the HTTP packets will only have headers, no body payload). This is negligible compared to the worker instance cost.

Content-based storage for deployments

I’ve introduced content-based storage in a previous blog post. The general idea is to identify data by its hash (often SHA-1 or SHA256) resulting in automatic deduplication and verifiable referential consistency. It is an ideal concept for versioning, that’s why it is also broadly used in the popular git distributed version control system. It is also an ideal approach for managing our deployments. Like this:

Think of assemblies, config and settings as files, deployments as commits (pointing to one assemblies, config and settings blob each, shared if equal), and HEAD as head just like in git. All the arrows include the full hash of the target (as part of their name, shortened in the diagram).

The Index is just a redundant list of all deployments for easier management so we don’t have to iterate through all available deployments all the time. In a similar way a History blob could be interesting to track the last few deployments and when they have been deployed. Note that HEAD and Index (and History) are the only mutable blobs, all the others are readonly, although they can be garbage collected.

The arrows between deployments will likely be dropped, they don’t seem provide any value in practice.

Prepare deployment, then activate atomically

Both the creation of a deployment (based on assemblies, config and settings) and actually activating it (by changing HEAD to point to it) are now atomic operations. They still can happen at different times though, so you can prepare one or more deployments but activate them much later, if at all. The applications can be completely unrelated, so you can use this mechanism to quickly switch between different applications.

Note that it still takes a while until all workers have detected the change, so there will be a phase where different workers (on different VMs and servers) may have different applications running. There are ways to deal with this if it is an issue, see below.

Get back to the previous version

… is as trivial as looking up the previous version in the History or Index and change HEAD to point to it.

Changing service settings

If you change some settings in the web console, for example disable a service or change a trigger interval, a new settings blob will be created, then a new deployment referring to the new settings, and in the end HEAD will be changed to point to the new deployment. If you decide to change it back, the new settings will already exist (with the same hash), so in effect only the HEAD blob will be changed back to point to the previous deployment (plus the History updated if available). Note that you won’t see much of that in practice as the management classes will handle it automatically.

Handling changes in the runtime

If the runtime detects a changed HEAD it will immediately load the deployment blob. Since it knows its current deployment and since the blobs are named after their content hash and readonly, it can simply compare the names to detect what blob has been changed. If either assemblies or config has changed, the runtime will have to restart all the processes, but if only the settings changed then it’s usually enough to just adapt the scheduling appropriately. Settings changes therefore still have a rather small impact in practice, despite switching to a completely different deployment in the storage.

Forcing a deployment form an application

Sometimes you need to ensure that a message is processed by a specific deployment. For example, we sometimes deploy a new version and then want to do some computation on exactly that version. To achieve that we could either wait, or include the deployment hash in the message and make the application force the runtime to load exactly that deployment if it isn’t matching already. For that and similar purposes I suggest to provide some way for services to send commands to the local runtime, like commands to enforce loading of the head or some specific deployment as soon as possible.

Multi-Head Scenario

This is unrelated to deployments, but the new runtime will also support multiple processes (”cells”) in parallel, isolated in separate AppDomains and threads and with independent scheduling. Services settings contain a new cell affinity parameter to control in what cells a service should be executed. This can be useful e.g. to create a cell for low latency queue services or to avoid blocking when some services can have long processing times but there are only few worker instances available.

Now technically it would be possible to load different applications or versions in different cells at the same time, with separate HEADs for each cell. This would bring the service interleaving approach to a new level. Not sure how useful it would be in practice though (plus it would require some work to decide which app can choose the number of worker instances), so I won’t follow that idea any further for now.

Too complicated?

This all seems very complicated just to do deployments. How could we simplify it but still satisfy our requirements?

  • Single blob only:
    Store everything (assemblies, config, settings) in a single blob and give the currently active blob a special name (like “current” or still “HEAD”). Way simpler. Disadvantage: the whole blob can get large and has to be touched by every single settings change. Unlikely an issue in most deployments though.

  • Drop the hashing:
    SHA is built in, so this is not really a big simplification. Disadvantage: we’d loose deduplication

  • Toggle instead of versioning:
    Just provide two versions of each blob, which can be switched on demand (similar to Azure staging vs. production deployments). The staging blobs would be edited and when done switched somehow in an atomic way (e.g. HEAD blob pointing to version again). This may simplify management remarkably.

  • Outsource the versioning:
    Use a proofed version control system instead, like git. Technically even subversion would work (I’ve tested subversion on Azure in the past, worked fine. For git we could even use one of the native git libraries). Disadvantage: checking a remote repository for changes is more expensive than a simple azure blob storage ETag check (git beats svn here). In the worst case we could work around that by introducing a HEAD blob in azure storage again, containing the current head revision/hash. We would then update that blob after every commit and poll it from the workers at much higher frequency. Advantage: Much more robust versioning, we could drop the zip files (simply version the assemblies directly), and we’d get push deployment for free.

Personally I like that last alternative using git the most.

Feedback

What do you think? Too complicated? Overengineered? Schould I use a real git repository instead? Let me know. Thanks!

(Migrated Comments)

Rinat Abdullin, July 23, 2011

Once again, that’s a fine post, just like the previous one on hashing. I loved rereading it.

Just a few thoughts.

  1. How hard would be to use Lokad.Cloud cell management with AppDomains without using actual services and message dispatch?
  2. I think that given versioning sandbox/production toggle is just an overkill (it duplicates the logic). swaps are just a way to shift focus from one version to another, while keeping the ability to roll-back.
  3. Hashing and separated blobs, as I believe, are a must for a simple implementation. They allow to keep stuff simple and decoupled. Besides, the complexity could be reduced by the tooling.
  4. Git (versus self-implemented versioning) is just a way to deliver changes in my opinion (besides, versioned settings). So it should not be that different for blob or git storage (we just poll for changes and pull the version specified by head). I’m wondering how well would private github repo work here…

In case of blob, head is stored in blob storage (human-editable JSON), pointing to the deployment blobs In case of full git, head is in git In mixed scenario, head is stored in blob storage, pointing to the git version/url

Sorry for pushing that much of my rambling here :)

]]> <![CDATA[Content-Based Storage in the Cloud]]> 2010-07-21T12:19:00+02:00 http://christoph.ruegg.name/blog/2010/7/21/content-based-storage-in-the-cloud One derivative of the NoSQL movement that rediscovers non-relational storage approaches lately is a content-based value store. Such a store is similar to a Key-Value store but uses a cryptographic hash of the value as key.

An SHA-1 hash of the value is good enough to identify it

The SHA-1 hash function is unique, meaning that for every value there’s exactly one key that can be computed using SHA-1, hence value implies key. We can always compute the unique key of a value.

The probability of an SHA-1 hash collision is extremely low. The most cited numbers show that you’d need 1024 values in order to cause a 50% chance of a collision. Even with a whopping 1018 distinct values the likelihood of at least one collision is already down at 10-9. Hence, a key refers to a single value with extremely high probability. While the SHA1 function is not strictly injective, it is approximatively injective enough for almost all practical applications.

Note that this is different from common non-distributed Hash Tables where a very short hash function is used to directly jump fast to an inner data structure (bucket) containing all items sharing the same hash. The motivation for hashing in such hash tables is to be able to directly compute the position where an item is stored, avoiding long linear or binary searches.

Verifiable Consistency

A nice side effect of using SHA1 is that given the key, the value retrieved from the storage can be verified (detect data corruption or tampering) simply by recomputing its SHA-1 hash and comparing it to the key. You can even digitally sign a key and by that implicitly sign its value and all those referred by it.

Keys are uniformly distributed

When using the common hex string format, the 160bit long keys are always 40 characters long and look something like this:

d921970aadf03b3cf0e71becdaab3147ba71cdef

We can safely treat them as if their characters were distributed uniformly (0-9, a-f). This brings some advantages especially when used in a distributed or cloud-like scenario, as simple prefix ranges (like 0-3, 4-7, 8-b, c-f) can be used for partitioning and distributed processing.

On the other hand this means that you can’t use other indexed keys or ordering out of the box without further logic or storage on top of it.

The value of a key is fixed and can’t be changed

The value associated with a key can never change. If you store an updated value, you’ll get a new key for it and update the reference to this new key. This has severe consequences on where this storage scheme can be used efficiently. For example, a typical relational data model with cyclic relations wouldn’t fit at all to such a content-based data store.

However, in practice in a cloud-like application this is often not that much of an issue. Even more so as soon as you realize that the existence of read-only stale yet still consistent data is not an issue either (see CQRS).

Again this fits very well with distributed and cloud computing, as it becomes trivial to aggressively cache values locally. If a value is found in the local cache it is guaranteed to be up to date (since values can’t change), so you don’t even have to check for timestamps or whether it has been changed remotely. Since in Azure the instances come with a lot of local storage, a simple MRU cache for a few GB can save you a lot of downloads and roundtrips if you use only a relatively small number of instances or have managed to create some weak affinity between jobs and Azure instances.

Example: Large Queue Messages

Azure Queues have content size limitations, that’s why Lokad.Cloud implements logic to let messages transparently overflow to blob storage. To do that it needs a way to store a value in a blob that it can retrieve later by some identifier. This identifier is then packed to the actual message. There’s no need to access it in any other way, so it’s a perfect candidate for a content-based value store.

In my experience, in real life the probability that a message is processed on the same worker that originally put it there is often high or at least not negligible. In all these cases, a cached content-based value store would save you from having to download these blobs completely, but still work correctly otherwise.

Implicit Value-Deduplication

Since the same value leads to the same key, trying to store the same value twice means you get the same storage location and the value gets stored only once. The second trial can even be aborted early by provoking a precondition violation, or skipped completely if it is already in the local cache (depending on the deletion plan).

Example: Daily Backup Snapshots

I recently wrote a small service that periodically takes full snapshots of all tables and blobs of a set of Azure storage accounts to a separate account, keeps the last N snapshots each and removes the rest. Often only a small subset of blobs or table entities actually change in a day. Had I used content-based storage, I could have saved a lot of storage (and thus cost) by deduplication without having to implement complicated incremental or differential backups. Taking a snapshot would likely also have taken less time thanks to some saved uploads.

Trivial Distribution and Replication

Other than any classical relational databases and key value stores, replication and distribution of data in such a content-based value store is trivial since there can’t be any conflicts. This is why the caching mentioned above works so well. Replication simply means to copy the values of all missing keys over to the target. A consequence of this is that for some scenarios there’s no technical need for a single master database. A peer can synchronize with any other peer, resulting in full peer to peer support. Distributed hash tables (DHT) as used by most file sharing solutions including BitTorrent work similarly and turn out to be very efficient.

History Consistency and Versioning

Since values can’t change, they remain consistent with each other even when they become stale. That’s why this approach is used by most of the popular distributed version control systems like Git and Mercurial as well.

The Git object model is nicely described in the git community book (the following two images are taken from there). In essence, all objects are stored just as described here. In addition to data blobs (i.e. source code files) there are also tree objects representing a folder simply by listing all the SHA-1 keys of its child elements, again stored by its hash:

If a file changes in git in a new revision, it will get a new hash. The folder/tree containing it will update that hash in its list, and in turn will itself get a new hash. Both the old a new version are therefore still available completely and consistently simply by referring to the hash of the respective version of the tree.

Historical consistency can be useful for all kind of applications. Note that this approach persists snapshots of values and content, not how they are changed. This is thus a dual counterpart to concepts like event sourcing where only the actions causing changes of the values are persisted but not the actual values.

Append-Only Storage or Value Scavenging

Unless you need an append-only storage, you need to be careful about deleting values in such a system. Since there is implicit deduplication, you can’t just delete what you’ve just inserted since the same value could also be used in other places. There are several approaches how you can attack this, depending on your scenario:

  • Garbage Collection: If there is a hierarchy where all values are referenced by another value, you can follow the tree from time to time and then remove all values you haven’t seen. This is used by all the distributed version control systems. Be careful about race conditions though.

  • Reference Tracking: Use metadata to list all keys or items referring a value. If you remove the last reference, remove it. This can be combined with garbage collection. You can also use reference counters, but they are difficult to handle correctly in an unreliable world like a cloud environment where instantaneous VM shutdowns without prior notice are to be expected.

  • Time-Based: You “touch” a value (update a timestamp in the metadata) whenever it is used, and from time to time remove all items that haven’t been used for a while. Note that that causes a lot of round trips (although they could be performed asynchronously in the background).

  • Limited Lifetime: Sometimes its good enough to just define that a value can safely be removed after a day or a month.

]]> <![CDATA[How to create 2048bit Certificate CSRs for Dell's iDRAC6]]> 2010-07-20T11:21:00+02:00 http://christoph.ruegg.name/blog/2010/7/20/how-to-create-2048bit-certificate-csrs-for-dells-idrac6 In case you happen to manage a recent DELL server with a dedicated iDRAC remote management card and you’d like to secure it by using your own certificate, you’ll have to request a certificate based on a CSR request created directly in the iDRAC web interface.

Unfortunately these CSRs have only 1024 bit keys, which get refused by some public certificate authorities like StartCom (for security reasons they require at least 2048 bits). You can’t choose the bit length in the iDRAC web interface, but luckily there is another way to make it generate 2048 or 4096 bit long keys for the CSR using racadm from Dell’s System Management Tools:

View the current configuration (all on 1 line):

racadm.exe -r [iDRAC IP] -u [user] -p [password]
getconfig -g cfgRacSecurity

Change the key length to 2048 bits (all on 1 line):

racadm.exe -r [iDRAC IP] -u [user] -p [password]
config -g cfgRacSecurity -o cfgRacSecCsrKeySize 2048

(Migrated Comments)

Dan Orum, September 7, 2010

If you are using the Express version of the iDRAC card, you can’t use the racadm.exe utility with an IP address remotely. Instead, you need to run the utility on the local server without specifying the -r parameter.

Christoph Ruegg, September 11, 2010

Indeed, thanks for the hint!

]]> <![CDATA[git howto: revert a commit already pushed to a remote repository]]> 2010-05-05T23:22:00+02:00 http://christoph.ruegg.name/blog/2010/5/5/git-howto-revert-a-commit-already-pushed-to-a-remote-reposit So you’ve just pushed your local branch to a remote branch, but then realized that one of the commits should not be there, or that there was some typo in it. No problem, you can fix it. But you should do it rather fast before anyone fetches the bad commits, or you won’t be very popular with them for a while ;)

Case 1: Revert the last commit

This is the easiest case. Let’s say we have a remote mathnet with branch master that currently points to commit dd61ab32. We want to remove the top commit. Translated to git terminology, we want to force the master branch of the mathnet remote repository to the parent of dd61ab32:

$ git push mathnet +dd61ab32^:master

Where git interprets x^ as the parent of x and + as a forced non-fastforward push. If you have the master branch checked out locally, you can also do it in two simpler steps: First reset the branch to the parent of the current commit, then force-push it to the remote.

$ git reset HEAD^ --hard
$ git push mathnet -f

Case 2: Revert the second last commit

Let’s say the bad commit dd61ab32 is not the top commit, but a slightly older one, e.g. the second last one. We want to remove it, but keep all commits that followed it. In other words, we want to rewrite the history and force the result back to mathnet/master. The easiest way to rewrite history is to do an interactive rebase down to the parent of the offending commit:

$ git rebase -i dd61ab32^

This will open an editor and show a list of all commits since the commit we want to get rid of:

pick dd61ab32
pick dsadhj278
...

Simply remove the line with the offending commit, likely that will be the first line (vi: delete current line = dd). Save and close the editor (vi: press :wq and return). Resolve any conflicts if there are any, and your local branch should be fixed. Force it to the remote and you’re done:

$ git push mathnet -f

Case 3: Fix a typo in one of the commits

This works almost exactly the same way as case 2, but instead of removing the line with the bad commit, simply replace its pick with edit and save/exit. Rebase will then stop at that commit, put the changes into the index and then let you change it as you like. Commit the change and continue the rebase (git will tell you how to keep the commit message and author if you want). Then push the changes as described above. The same way you can even split commits into smaller ones, or merge commits together.

]]> <![CDATA[Lost in Math.NET Codenames?]]> 2010-04-26T20:27:00+02:00 http://christoph.ruegg.name/blog/2010/4/26/lost-in-mathnet-codenames Math.NET Numerics? Iridium? dnAnalytics? Yttrium? Huh? …sounds familiar?

It looks like some of you got lost in all the Math.NET subprojects and codenames. Math.NET evolved over time, with projects splitting into separate new projects, the introduction of codenames and new projects replacing older ones with a slightly different focus and approach. Unfortunately this lead to a mess (sorry for that!), so I’m trying to throw light on it by the following small chart, depicting the Math.NET Project history:

It all started with MathLib which was a very verbose object oriented computer algebra approach, including all kind of numeric routines to back the symbolics, including basic linear algebra. At the same time dnAnalytics was founded independently and unrelated to Math.NET, focusing entirely on numerics and statistics, leveraging highly optimized native libraries for better performance.

Soon it became obvious that it would make sense to refactor out the numerical aspects of MathLib to a separate project and to develop it independently, so Numerics was born, as well as several other non-numeric subprojects. Numerics became Iridium, and in 2009 Iridium and dnAnalytics finally decided to join forces and work together on the new Math.NET Numerics project, replacing both Iridium and dnAnalytics and entirely unrelated to the early Numerics 0.1-0.4 back in 2004.

Mostly thanks to Marcus and Jurgen, Math.NET Numerics is very well alive and active. Check out our source code repository and forums.

]]> <![CDATA[Connect from Azure to an SQL Server Named Instance]]> 2009-12-23T17:52:00+01:00 http://christoph.ruegg.name/blog/2009/12/23/connect-from-azure-to-an-sql-server-named-instance In some situations you can’t or don’t want to move all your data completely to the cloud. Be it to connect to your existing infrastructure, a company policy, to remain multi-tenant or simply when migrating slowly step by step. Common to these cases is often the requirement to synchronize with or connect from Azure to some local or offsite SQL Server database. For synchronization you may want to try the Microsoft Sync Framework. This post is about the other option: connecting to an external named SQL Server instance.

Connecting to Named SQL Server Instances

In addition to its own storage options like SQL Azure and Azure Table Storage, Azure also allows you to connect to external SQL Servers over TCP/IP. However, there’s a pitfall right now when using named SQL Server instances:

System.Data.SqlClient.SqlException:
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.
(provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
source

Provided your connection string is correct, this is likely to be an issue with how SQL server finds your named instance.

Instance Resolution using SQL Server Browser

Since SQL Server 2005, the SQL Server Browser service is responsible for enumerating available instances on a machine, and to resolve instance names to the actual named pipe or TCP port (for SQL Server 2000 it was the SQL Server Resolution Protocol).

In order to resolve the TCP port of a named instance, the client sends an UDP datagram to port 1434, to which the server browser replies with another datagram listing the instance endpoint to which the client then connects to. Thanks to this mechanism it is no longer required to have the server listen on the standard SQL server TCP port 1433, so it can fully support multiple (named) instances. In fact, the default for named instances is to use a dynamic random TCP port.

Azure vs. SQL Server Browser

When connecting from Azure this resolution mechanism fails, simply because the UDP datagrams never reach their target (this may change in the future). So there’s no way the client can find the actual probably random TCP port to connect to, and will throw the SqlException cited above.

Solution

To work around this issue, you can configure your named instance to listen on a static TCP port instead of randomly selecting a new dynamic one on every restart (related kb). You can then specify this static port directly in the connection string in your Azure worker role:

Data Source={domain/ip},{port};Network Library=DBMSSOCN;
Initial Catalog={dbname};User ID={user};Password={pw}

Note that in this case there’s no need to specify the name of the instance in the connection string. The network library parameter tells the client to use TCP/IP instead of e.g. Named Pipes.

]]> <![CDATA[Azure: Cloud Service Models]]> 2009-12-15T13:26:00+01:00 http://christoph.ruegg.name/blog/2009/12/15/azure-cloud-service-models Since I joined Lokad this September I finally had the chance to dive into cloud computing. We chose Windows Azure as platform for our very computation intensive business, and built a neutral opensource framework on top of it: Lokad.Cloud.

Cloud Services

Lokad.Cloud is described as a .net object-to-cloud persistence mapper, but it’s actually much more. This post shall concentrate on one aspect only: Its notion of Cloud Services as horizontally scalable workers.

In essence, cloud services are managed and executed as follows:

  1. The Lokad.Cloud management infrastructure (for now essentially a web role) allows you to upload one or more assemblies containing a set of cloud services and optionally some configuration file.

  2. Every Azure worker role instance loads all these services in an isolated AppDomain.

  3. Each Azure worker then executes these services one at a time according to some scheduling algorithm and execution policy.

We provide specialized base classes to simplify implementing services processing items from a shared queue or for services which are to be called in regular intervals.

We treat all azure workers as equal and therefore execute every cloud service on each Azure worker from time to time. In other words, we map all cloud services to all Azure workers, forming a complete bipartite graph between cloud services and Azure workers as shown in the following figure.

This is a fundamental concept that yields a very simple design with a potential for ideal horizontal scaling, and is even resilient to failing azure workers as long as at least one worker remains intact.

Cloud Service Models and Deployments

The only object that is aware of this mapping is the service scheduler. Yet, from the management and diagnostics perspective it would be interesting to represent the cloud services as first class objects. I’m therefore introducing the notion of Cloud Service Models for Lokad.Cloud (not part of the current release, open whether it ever will be).

In Azure, web and worker roles are explicitly defined and configured in two xml files. Since the latest update of the Azure tools for Microsoft VisualStudio, they are referred to as Azure Service Model. Using the Azure management website one can upload an assembly plus the two xml files to create a unique Azure deployment. A deployment can be stopped or running, either in production or in staging mode.

The same concepts can also be applied to Cloud Services, on a slightly higher level of abstraction and orthogonal to the Azure terms.

A Cloud Service Model is a unique entity, associated with a set of assemblies, the cloud services defined in them and their configuration (if applicable). Using the Lokad.Cloud management tools an administrator can upload such a model and create a unique Cloud Service Deployment. A deployment can be stopped or running, and of course be removed when no longer needed. A failing or malfunctioning deployment can be diagnosed and dealt with directly in the management UI.

Note that the currently implemented option to upload a zip file containing assemblies and optional configuration is already very close to such a models, but is missing identity and other metadata.

In each Azure worker, our scheduler will load the current service model, load the services and schedule them accordingly. From time to time the scheduler will check whether the deployed service model has changed, and update if necessary.

Technically this design would also allow to run multiple different deployments in parallel, e.g. by breaking the complete bipartite graph between Cloud Services and Azure workers into a non-complete bipartite one where Azure workers are assigned to a single Cloud Service Deployment:

Or by sharing the Azure workers by Cloud Service Deployments in a way or another (e.g. in parallel, or round robin):

Remember however that some of these scenarios violate the fundamental concept mentioned above. Hence, as usual, there’s a tradeoff between flexibility and robustness.

Update

It seems there’s a better way to differentiate between cloud service models and deployments:

  • Model: An identity, a set of (named) cloud services, their assemblies and optionally some configuration.

  • Deployment: An identity, a set of models and their mapping to (Azure) worker nodes.

I.e. only one deployment can run at at time, but there’s an option to support configuring multiple models in a deployment. Also, there’s a trivial empty deployment where no models are loaded at all.

Hence, the labels in the figures above should read “Cloud Service Model A” instead of “Cloud Service Deployment A”, etc.

]]> <![CDATA[dnAnalytics + Iridium = Math.NET Numerics]]> 2009-08-03T10:44:00+02:00 http://christoph.ruegg.name/blog/2009/8/3/dnanalytics-iridium-mathnet-numerics You may have wondered why the Math.NET Iridium development has stopped abruptly almost two months ago. Luckily this is not entirely true, in the last few weeks the .Net numerics library has progressed well - but at a different place:

Math.NET Iridium is being merged with dnAnalytics, resulting in a new project named Math.NET Numerics

What does that mean for existing Math.NET Iridium users?

  • Higher development momentum and larger user community (as a direct result of merging two projects).

  • Better algorithm and code quality by picking the best of each project and simply by having new highly skilled developers on board.

  • New opensource license model: MIT/X11. This is a very open license similar to the so called New BSD License. This model is much less restricting than the previous LGPL and is (to my knowledge) source-compatible to a wide range of licenses including all GPL-based licenses and the Microsoft opensource licenses, too.

  • Some API changes. This is unavoidable since we try to integrate the best of both dnAnalytics and Iridium. At the same time this is a good chance to throw out some old designs that have shown to be improvable and replace them with better approaches. However, we try hard to keep migration as smooth as possible.

  • In addition to the completely self-contained managed implementation, we’ll profit from the dnAnalytics experience with parallelized and native optimizations (MKL, ACMS, CUDA etc) and will therefore provide optional wrappers around native libraries which provide significantly better performance when working with large data sets.

  • Again thanks to the dnAnalytics experience, you can expect better F# support, even though the library is still written in C#.

  • Although Iridium did support sparse linear algebra for a very short time, we had to remove it due to several issue. You can expect Math.NET Numerics to finally support sparse linear algebra in a clean way.

You’ll find the new Math.NET Numerics discussion board and tracker at CodePlex and the current sources at Github (subversion mirror at google). The full portal website and wikis etc. will be available in a few weeks. Feel free to post your ideas, feedback or even fork the repository at github to contribute code to the project (note that we will completely reorganize the project structure until mid August).

We’ll let you know here and on Twitter as soon as we reach a first milestone and have an api preview ready.

(Migrated Comments)

Joannes Vermorel, August 3, 2009

Congratulations! Sparse linear algebra is really a nice move (I am sorry I had not been able to push it forward at the time).

Alexey Zakharov, October 23, 2009

Good news! C# really needs such library in stable version.

]]> <![CDATA[Online API Reference]]> 2009-04-17T18:08:00+02:00 http://christoph.ruegg.name/blog/2009/4/17/online-api-reference We now finally provide an online api reference in an rdoc-like style, generated by docu (actually by my github fork of it). Note that docu is new and still under heavy development, so the quality is likely to improve over the next months (e.g. right now the class summaries are missing).

http://api.mathdotnet.com/

It is simple, but (other than the older NDoc & Sandcastle generated sites) loads very fast.

]]> <![CDATA[Iridium Statistics Accumulator: Better numerical stability]]> 2009-01-07T21:12:00+01:00 http://christoph.ruegg.name/blog/2009/1/7/iridium-statistics-accumulator-better-numerical-stability The algorithm on how the Mean, Variance and Sigma are incrementally computed in the statisics accumulator (MathNet.Numerics.Statistics.Accumulator) has been improved last week in Iridium revision 503 to provide better numeric stability when dealing with samples with a very large mean but only a small variance.

For example, the variance of normally distributed samples with mean 10e+9 but a variance of only 1 can now be accurately estimated. The previous implementation has been very unstable in that case.

The new algorithm continues to support removing samples from the accumulator (and updates the estimates accordingly).

]]> <![CDATA[Iridium 2008 August Release (2008.8.16.470)]]> 2008-08-14T14:14:00+02:00 http://christoph.ruegg.name/blog/2008/8/14/iridium-2008-august-release-2008816470 Iteration 16 of Math.NET Iridium (Numerics) is now available, as 2008 August Release with Version 2008.8.16.470. Grab it here.

Please continue reporting issues and bugs you find, it’s very useful and helps making the whole project better. We’ve also setup a UserVoice page for you to suggest or vote for new features or enhancement.

Team: Christoph Rüegg, Joannès Vermorel, Matthew Kitchin

Summary

  • Bugs: 4 bugs have been fixed.
  • Completely revised and extended interpolation toolkit.
  • New complex matrix and vector type (Complex Linear Algebra will follow in the next iteration).
  • Slightly enhanced real matrix and vector types.
  • QR decompositions are now unique (positive real R diagonal).
  • Complex type now has a public constructor, more intuitive.
  • New Digamma (Psi) special function.
  • Various other small changes.

For more details have a look at the download page.

(Migrated Comments)

Joannes Vermorel, August 14, 2008

Thanks for this good work. I have also noticed that you’ve posted the release on Sourceforge.Net. This is really a nice thing to do (so far, I have always found Sourceforge to be a top notch backup provider :-)

Christoph Ruegg, August 14, 2008

Thanks. Yes, indeed, the SourceForge mirror infrastructure is unbeatable :)

]]> <![CDATA[Revised Interpolation Toolkit in Iridium]]> 2008-05-10T17:31:00+02:00 http://christoph.ruegg.name/blog/2008/5/10/revised-interpolation-toolkit-in-iridium The next release of Math.NT Iridium, Iteration 16, comes with a revised interpolation architecture and implementation.

Up to now, the interpolation classes have been a bit awkward to use, partially because of the SampleList collection class you had to use, and because of the design in general. It also provided only two interpolation algorithms, which are both somewhat outdated these days.

The new implementation provides some newer more stable algorithms (like Floater and Hormann’s algorithm for pole-free rational interpolation) together with a cleaner design. Additionally, some of the algorithms can also provide the first and second derivative and in the case of splines even a definite integration. There is also a new facade/portal class that reduces building an interpolation to one simple method call. Usually you would just use this facade class to build/precompute an interpolation, but all the algorithms are also publicly available in the Algorithms-namespace, so if you know what you’re doing you can use them directly.

Sample code, which uses a pole-free rational barycentric interpolation (the default algorithm) with 5 given sample pairs (t, x(t)):

1
2
3
4
5

double[] t = new double[] { -2, -1, 0, 1, 2};
double[] x = new double[] { 1, 2, -1, 0, 1};
var method = Interpolation.Create(t, x);

double res = method.Interpolate(0.5);

Simple, isn’t it?

Unfortunately it was not possible to fit the new design into the old classes, so the new classes and interfaces replace the old classes completely. These old classes are still there for now and continue to work, but they’re marked as obsolete and we recommend strongly to upgrade your code base to the new architecture.

The new architecture has already been checked in to the source repository. If you’re interested, please have a look at it and provide feedback - it’s not released yet so we can still change it completely :).

]]> <![CDATA[Iridium 2008 April Release (v2008.4.14.425)]]> 2008-04-06T13:50:00+02:00 http://christoph.ruegg.name/blog/2008/4/6/iridium-2008-april-release-v2008414425 Iteration 14 of Math.NET Iridium (Numerics) is now available, as 2008 April Release with Version 2008.4.14.425. Grab it here.

Sorry for the very short release cycle (just one week after iteration 12). The reason is that I won’t be able to work on Math.NET for the next three weeks and that some of the fixes and changes are important enough to not let you wait three weeks for no reason.

Please continue reporting issues and bugs you find, it’s very useful and helps making the whole project better. There’s also a big chance that the issue will actually be fixed: in the last few releases we always managed to fix all bugs we were aware of at that point. Thanks!

Team: Christoph Rüegg, Joannès Vermorel, Matthew Kitchin

Summary

  • Bugs: All known 3 bugs have been fixed.
  • Better special function precision (Gamma, Beta, Erf, Distributions etc): now up to 12 - 14 digits.
  • New direct/real gamma function, new harmonic number function.
  • Interpolation: usability enhancements (better double-array support, less user code)

New Features

  • IRID-122: Core - New direct Gamma function (additional to GammaLn) with negative value support
  • IRID-123: Core - New Special Function: Harmonic Numbers

Ehancements

  • IRID-121: Core - Better numerical precision for Gamma function
  • IRID-125: Interpolation - Additional interpolation and sample list constructors for double arrays.
  • IRID-126: Interpolation - Better interpolation order access and defaults

Fixed Bugs

  • IRID-119: Interpolation - Polynomial Extrapolation in positive direction throws IndexOutOfRangeException
  • IRID-120: Linear Algebra - Infinite recursion
  • IRID-124: Linear Algebra - Matrix.CopyToArray - wrong indexer in inner loop condition.
]]> <![CDATA[Iridium 2008 March Release (v2008.3.12.405)]]> 2008-03-31T11:41:00+02:00 http://christoph.ruegg.name/blog/2008/3/31/iridium-2008-march-release-v2008312405 Iteration 12 of Math.NET Iridium (Numerics) is now available, as 2008 March Release with Version 2008.3.12.405. Grab it here.

Team: Christoph Rüegg, Joannès Vermorel, Matthew Kitchin
New Contributions: Mike Shugai

Summary

  • Bugs: All known 3 bugs have been fixed.
  • Linear Algera: Matrix Kronecker product, new Vector class, performance work (caching).
  • Core: New Sinc function, Neper/Decibel ratio routines.
  • Probability Distributions: New distributions: Student’s T-Distribution, F-Distribution, Skew-Alpha Stable Distribution.
  • Assemblies no longer signed with a certificate (because verification caused network access).

New Features

  • IRID-113: Core - Neper and Decibel Helper and Factors
  • IRID-116: Core - Sinc Function
  • IRID-111: Linear Algebra - Matrix Kronecker Tensor Product
  • RID-59: Linear Algebra - New Vector class (related to the Matrix class)
  • IRID-99: Probability Distributions - New Distribution: Skew Alpha Stable Distribution
  • IRID-108: Probability Distributions - New Distribution: F-Distribution
  • IRID-109: Probability Distributions - New Distribution: Student’s-T

Enhancements

  • IRID-106: Linear Algebra - Cache for on-demand computations (like decompositions)

Fixed Bug

  • IRID-107: Core - Complex: Unexpected power behavior at zero
  • IRID-97: Linear Algebra - Matrix.Identity allocation bug in non-square cases
  • IRID-98: Probability Distributions - ArbitraryDistribution NextInt32 does not consider offset.

Other

  • IRID-110: Remove Certificate Signing (cert validation causes network access)

(Migrated Comments)

Joannes Vermorel, March 31, 2008

Nice to see that new people are joining Math.NET. If I get the time, I will try to submit some Erlang-related formulas and distribution for the next release :-)

Christoph Ruegg, April 6, 2008

Thanks, I’m looking forward to it! :)

]]> <![CDATA[Iridium 2008 February Release (v2008.2.10.364)]]> 2008-02-03T12:39:00+01:00 http://christoph.ruegg.name/blog/2008/2/3/iridium-2008-february-release-v2008210364 Iteration 10 of Math.NET Iridium (Numeric) is now available, as 2008 February Release with Version 2008.2.10.364. Grab it here.

This is mostly a service release.

  • Bugs: All known bugs have been fixed.
  • Performance: The linear algebra implementation has ben optimized, resulting in nearly 50% perf gain.
  • Api References: Inline Xml Documentation has been improved (but is still far from where we’d like it to be).
  • Security: The released binaries now have a strong name, are locked down with code access security and allow partial trusted callers. We now use test-signing internally. Also, the official assemblies are now signed with a certificate.
  • Build/Release Integration: We now moved completely to custom msbuild targets, releases are now fully automated (incl. documentation generation) and the continuous integration system has been upgraded. Since releasing is now much easier, you can expect new releases more often.

At the same time I also released a first version of Math.NET Neodym (Signal Processing), Iteration 2 what makes it Version 2008.2.2.364. Grab it here. Hopefully I’ll have more time in the future to work on Neodym…

(Migrated Comments)

J.Henkel, February 6, 2008

Excellent. Time to play a bit…

Out of curiosity, Chris, are you planning to release a build of Yttrium any time soon? I’ve been playing around with a copy from svn, and just reading through the test files makes me want to play with a compiling version (mine is old…keeps looking for MathNet.Numerics.Fn.PowInt).

]]> <![CDATA[Matrix data structure optimization: Nearly 50% perf gain]]> 2007-10-14T21:28:00+02:00 http://christoph.ruegg.name/blog/2007/10/14/matrix-data-structure-optimization-nearly-50-perf-gain Some time ago I did some very basic performance analysis of solving linear equation systems with Iridium. Since then we decided to rewrite the Matrix class to use jagged arrays instead of rectangular ones, see this forum discussion. There already was a discussion about that issue some long time ago, but at that time we decided to go for the more clean and safe way of rectangular arrays. Unfortunately the C# compiler today still can’t optimize loops on rectangular arrays as good as loops on jagged arrays. So we finally moved forward to jagged arrays, and indeed, we got a performance improvement (solving a linear equation system) by nearly 50%.

Unfortunately, the change of the data structure comes at a cost: The semantics of the two following members changes, as they now do deep-copies instead of using the data structure directly as internal data structure. Have a look at the mentioned forum discussion on why this might be an issue.

public Matrix(double[,] A)
public static implicit operator double[,] (Matrix m)

If you want to avoid deep-copying, e.g. for performance reasons, then use double[][] instead of double[,] to fill the matrix.

The changes are already submitted to the repository, and will be included in the next iridium release.

(Migrated Comments)

Joannes Vermorel, October 15, 2007

This is an excellent news. Thanks for upgrading the matrix to jagged array. The use of rectangular arrays was a (poor) choice of mine when I did initially port the JAMA package to Math.NET.

Among the other benefits that you gain with jagged arrays is the possibility to perform multi-thread operations on matrices (using BLAS).

Best regards, Joannes

]]> <![CDATA[Encrypt local mail folders and the desktop search index]]> 2007-08-25T20:10:00+02:00 http://christoph.ruegg.name/blog/2007/8/25/encrypt-local-mail-folders-and-the-desktop-search-index I’m using TrueCrypt to encrypt all kind of files on my computer (and on all external harddrives and most USB thumbdrives) for quite some time now and I really like it. On the PC I have a set of data I want to have encrypted but that I always need, so I’ve configured it to automatically mount it as X:\ after login (that is, I just have to enter my password, hit enter and forget about it). One of this kind of data is the web browser bookmarks, history and cache. Luckily Firefox allows you to store the profile wherever you want, so I simply store it on that virtual drive X mentioned before.

Desktop Search is very useful (be it Google Desktop or Microsoft Desktop Search or whatever). However, what’s the point of encrypting files when they’re full-text indexed to some unencrypted location anyway. Hence I want this index to be stored on that encrypted drive X too.

  • Issue: The Windows Desktop Search service starts before I can enter the password and mount X. If it can’t access the index, it automatically starts generating a new index at the default (and unencrypted) location. So I need some way of delaying the service start.

In addition I also want my local mail folder encrypted on X. Similar to Firefox, the Mozilla email client Thunderbird allows my to store the profile wherever I want, so I chose to store it directly on X. However, I also use Outlook 2007, and there it’s not that simple:

  • Issue: While Outlook 2007 allows you to store the main post folder wherever you want, for some yet unknown reason you can’t move IMAP or Exchange data folders.

Luckily there is a workaround for both issues, although it’s not trivial and involves some coding. Here’s what I did, let me know if you have a better solution (the NTFS built-in encryption is no option for me), or if you’re interested in the code:

  • Change the Search Index Service to start manually (instead of automatically)
  • Write a small windows service that starts automatically and checks all few seconds whether X is mounted. If it is, it stops further checking and starts the Search Index Service.
  • Move all files from the outlook data folder (something like Documents and Settings..\Local Settings\Application Data\Microsoft\Office) to a new folder in X and delete this Outlook folder
  • Have the small windows service (from point 2) create an NTFS Junction (aka symlink) in place of the just deleted Outlook folder, and point it to the new directory in X, as soon as X is mounted; and remove this junction again once the service stops. This makes Outlook 2007 think the files are still where it expects it, but in reality they’re on X instead.
]]> <![CDATA[Math.NET Iridium on Linux and Mono]]> 2007-08-19T18:43:00+02:00 http://christoph.ruegg.name/blog/2007/8/19/mathnet-iridium-on-linux-and-mono One of the principles of the Math.NET Iridium numerics library is to not depend on special hardware or some external super-optimized library, but just on the core .Net Framework 2.0. Thanks to this, Iridium also works with Mono on Linux with no special treatment - just add a reference to the assembly in MonoDevelop and start coding…

Apparently some users are not used to reference .Net assemblies on Mono yet, so I’ve created some screen shots showing how to add such a reference step-by-step in MonoDevelop on Ubuntu Linux:

Math.NET on Mono (and Linux)

]]>